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ABSTRACT 


A  properly  operated  Lessons  Learned  System  supports  Knowledge  Management 
and  Organizational  Learning.  The  method  of  handling  lessons  has  an  effeet  on  sueeessful 
operation  of  a  Lessons  Learned  System. 

This  researeh  evaluates  a  sample  of  Lessons  Learned  Systems  for  their  method  of 
handling  lessons.  It  provides  a  eoding  that  allows  a  Lessons  Learned  System  to  be 
eharaeterized  over  the  speetrum  of  possible  handling  methods.  It  relates  this  eoding  to  its 
effeet  on  the  three  tasks  of  a  Lessons  Learned  System;  eolleeting  lessons,  insuring  quality 
of  lessons  for  dissemination  and  dissemination  of  the  lessons  sueh  that  implementation 
oeeurs. 

This  method  allows  for  Lessons  Learned  System  evaluation  and  design  with 
respeet  to  the  handling  of  lessons. 
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I.  INTRODUCTION 


A,  PURPOSE 

This  thesis  has  three  purposes. 

The  first  is  to  increase  the  body  of  knowledge  that  exists  for  Lessons  Learned 
Systems.  As  resources  for  organizational  application  become  limited,  it  is  important  that 
those  resources  are  used  in  the  most  efficient  manner.  A  properly  operating  Lessons 
Learned  Systems  will  increase  the  efficiency  of  operation.  Further,  safety  of  operation 
may  also  be  increased  with  a  properly  operating  Lessons  Learned  System.  Documenting 
and  analyzing  existing  Lessons  Learned  Systems  will  provide  future  designers  of  Lessons 
Learned  Systems  a  resource  and  a  foundation  upon  which  to  build. 

The  second  purpose  is  to  focus  the  various  methods  of  lessons  handling. 

Handling  refers  to  the  level  of  treatment  given  a  lesson  after  it  has  been 

generated.! 

The  focus  is  to  provide  a  characterization  that  will  encompass  the  various 
methods  of  lessons  handling.  Further,  the  characterization  will  be  a  tool  for  the  design  or 
architecture  of  a  Lessons  Learned  System  by  connecting  the  characterization  to 
successful  operation  of  a  Lessons  Learned  System 

The  third  purpose  is  to  apply  the  characterization  of  lessons  handling  and  its 
relation  to  successful  operation  to  the  SUPSHIP2  Groton  Lessons  Learned  System.  The 
application  may  provide  recommendations  for  improvement. 


1  Snider,  K.  F.,  Barrett,  F.  J.,  &  Tenkasi  R.  (2002). 

2  Supervisor  of  Shipbuilding,  Conversion  and  Repair,  Groton,  CT. 
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B. 


BACKGROUND 


The  Defense  Systems  Management  College  defines  Lessons  Learned  as: 
Capitalizing  on  past  errors  in  judgment,  materiel  failures,  wrong  timing,  or  other  mistakes 
to  ultimately  improve  a  situation  or  system. 3  It  defines  a  system  as:  The  organization  of 
hardware,  software,  material,  facilities,  personnel,  data,  and  services  needed  to  perform  a 
designated  function  with  specified  results,  such  as  the  gathering  of  specified  data,  its 
processing,  and  delivery  to  users. 4 

A  Lessons  Learned  System  is  integral  to  any  organization’s  process  of  achieving 
its  full  potential.  Although  full  potential  may  have  a  different  meaning  to  different 
organizations,  all  would  probably  equate  achieving  full  potential  with  success. 

Success  comes  from  wisdom.  Wisdom  comes  from  experience.  Experience 

comes  from  mistakes. 5 

How  can  success  come  from  making  mistakes?  The  answer,  of  course,  is  to  learn 
from  mistakes  so  that  the  same  mistakes  are  not  repeated  again. 

Those  who  cannot  remember  the  past  are  condemned  to  repeat  it.6 

The  existence  of  a  Lessons  Learned  System  alone  does  not  guarantee  that 
mistakes  will  not  be  repeated.  The  Lessons  Learned  System  must  be  properly  designed 
in  terms  of  collecting  lessons,  processing  the  lessons,  disseminating  the  lessons  and 
follow  up  activities  to  insure  that  the  lessons  learned  are  properly  implemented. 

The  Lessons  Learned  System  must  also  be  appropriate  to  the  organization.  The 
success  of  any  system  is  dependent  on  proper  architecture.  One  aspect  of  proper 
architecture  is  that  the  system  provides  a  useful  purpose. 

3  Glossary,  Defense  Acquisition  Acronyms  and  Terms  (2001). 

4  Ibid. 

5  Maier,  M.  W.,  &  Eberhardt,  R.  (2000).  The  Art  of  Systems  Architecting.  (2"‘*  ed.).  Boca  Raton, 
London,  NY,  Washington  D.C:  CRC  Press.,  page  17 

6  Santayana,  George  (1863-1952),  Retrieved  2002,  from 
http://www.chesco.com/' artnab.sntavana.html 
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No  system  can  survive  that  doesn’t  serve  a  useful  purpose. 7 

Understanding  the  properties  of  a  Lessons  Learned  System  is  done  by  a 
methodology  that  is  used  to  understand  any  complicated  system,  the  methodology  of 
decomposition.  The  Lessons  Learned  System  is  broken  down  into  its  components  or 
working  parts.  Understanding  the  working  parts  and  their  relationship  to  one  another 
helps  in  understanding  the  collection  of  the  working  parts  that  is  the  system. 

An  earlier  decomposition  was  included  in  a  presentation  made  to  the  U.S. 
Department  of  Energy  Society  for  Effective  Eessons  Eearned  Sharing  Spring  2000 
Meeting. 8  In  that  presentation,  characteristics  of  a  Eessons  Eearned  System  were  named 
Contents,  Organizational  Type,  Process  Type,  Target  Process  Relation,  Dissemination 
Type  and  Recommendation. 

The  decomposition  was  further  refined  in  an  Acquisition  Review  Quarterly 
article.9  It  was  proposed  that  the  characteristics  of  a  Eessons  Eearned  System  be  grouped 
by  Eesson,  Operational  and  Organizational  factors.  These  three  main  characteristics  were 
further  sub-divided  into  sub-characteristics.  The  previous  characteristics  presented 
before  the  Society  for  Effective  Eessons  Eearned  Sharing  were  included  in  this  level,  as 
were  new  characteristics.  The  qualitative  value  for  each  characteristic  of  the  collection 
described  in  an  organized  fashion  a  Eessons  Learned  System. 

Listed  under  the  Operational  category  is  the  characteristic  called  Handling.  The 
qualitative  value  for  Handling  ranges  from  rigorous  to  open.  A  Lessons  Learned  System 
that  employs  rigorous  Handling  would  evaluate  each  lesson  in  a  formal  manner.  A 
possible  example  of  formal  Handling  might  be  a  determination  of  the  root  cause  of  the 
lesson.  Another  example  might  be  determining  where  or  to  who  the  lesson  should  be 
disseminated.  A  Lesson  Learned  System  that  employs  open  Handling  would  accept  any 
lesson  as  valid  and  disseminate. 


7  Hillaker,  Harry  (1989),  chief  architect,  General  Dynamics  F-16  Fighter,  as  stated  in  a  USC 
Systems  Architecting  lecture,  November  1989. 

8  Aha,  D.  W.  (2000). 

9  Snider,  K.  F.,  Barrett,  F.  J.,  &  Tenkasi  R.  (2002). 
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One  objective  of  this  research  is  to  analyze  the  Handling  characteristic.  It  is  to 
establish  a  listing  of  the  various  ways  lessons  are  “handled”.  It  is  to  provide  a 
characterization  of  these  handling  methods  and  determine  a  cause  and  effect  relationship 
that  can  be  used  for  Lessons  Learned  System  design  or  architecture. 

The  Supervisor  of  Shipbuilding,  Conversion  and  Repair,  Groton  CT  (SUPSHIP 
Groton),  a  Naval  Sea  Systems  Command  (NAVSEA)  Field  Activity,  is  the  Government’s 
on-site  design,  manufacturing,  and  construction/repair  management  team  for  submarines 
designed  and  manufactured  at  Electric  Boat,  Groton,  CT.io  As  part  of  its 
strategic/business  planning,  SUPSHIP  has  adopted  a  Balanced  Scorecard  format  for 
improvement.il  One  focus  area  is  “customer”.  One  strategy  to  improve  customer 
satisfaction  is  the  initiation  of  a  Eessons  Eearned  System. 

Another  objective  of  this  research  is  to  apply  the  results  of  the  cause  and  effect  of 
the  handling  characteristics  towards  the  design  of  the  SUPSHIP  Groton  Eessons  Eearned 
System. 


C.  RESEARCH  QUESTIONS 

The  primary  research  question  of  this  thesis  is:  How  may  a  Eessons  Eearned 
System  be  characterized  according  to  the  handling  of  lessons  processing,  and  how  may 
such  a  characterization  be  applied  to  Eessons  Eearned  System  design  or  architecture? 

The  subsidiary  research  questions  are  as  follows: 

1.  How  are  lessons  processed  or  handled  in  a  sample  of  Eessons  Eearned 
Systems? 

2.  What  are  the  effects  of  these  ways  of  lesson  processing  or  handling  on 
Eessons  Eearned  System  operation? 

3.  To  what  extent  will  the  ways  of  lesson  processing  or  handling  of  the 
SUPSHIP  Groton  Eessons  Eearned  System  support  successful  operation? 


10  SUPSHIP  Groton,  CT  (2002),  page  1. 

1 1  Ibid.,  page  3. 
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D,  BENEFITS  OF  STUDY 


The  benefit  of  this  research  is  to  increase  the  body  of  knowledge  that  exists  for  a 
Lessons  Learned  System.  It  provides  a  sampling  of  Lessons  Learned  Systems  and  a 
collection  of  handling  methods  for  those  Lessons  Learned  Systems.  It  provides  some 
benefits  and  consequences  of  these  handling  methods. 

The  benefits  of  this  research  also  include  one  method  to  characterize  the  handling 
of  lessons  by  a  Lessons  Learned  System.  It  also  allows  application  to  Lessons  Learned 
System  design  or  architecture. 

The  last  benefit  is  an  evaluation  of  the  design  of  the  SUPSHIP  Groton  Lessons 
Learned  System  with  possible  recommendations  for  improved  operations. 


E.  SCOPE  AND  METHODOLOGY 

The  scope  of  the  thesis  will  include:  (1)  identifying  numerous  Lessons  Learned 
Systems,  (2)  acquiring  information  on  the  operation  of  the  Lessons  Learned  Systems  with 
focus  on  the  handling  of  lessons,  (3)  determining  a  relationship  between  this  handling  and 
operations,  (4)  characterization  of  the  handling  in  an  encompassing  manner,  (5)  relating 
the  characterization  to  design  or  architecture  and  (6)  apply  the  relationship  to  the 
SUPSHIP  Groton  Lessons  Learned  System. 

The  nature  of  this  thesis  work  is  exploratory  in  nature.  It  is  based  on  a  sampling 
of  Lessons  Learned  Systems.  The  analysis  is  of  a  qualitative  nature  and  is  based  on 
empirical  evidence  or  shared  experiences  from  the  Lessons  Learned  System  sample. 
Theoretical  principles  of  Knowledge  Management  and  Organizational  Learning  provide  a 
map  to  explore  consequences  of  handling  methods. 

The  methodology  used  in  this  thesis  research  will  consist  of  the  following  steps: 

1.  Review  literature  for  Lessons  Learned  System  basics  including 
instructions  for  Government  Lessons  Learned  Systems. 

2.  Review  literature  in  the  areas  of  Knowledge  Management  and 
Organizational  Learning. 

3.  Conduct  an  Internet  search  of  existing  Lessons  Learned  Systems. 
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4.  Make  personal  contact  with  a  person  involved  in  each  Lessons  Learned 
System. 

5.  Provide  a  questionnaire  and  supplement  by  e-mails/phone  calls  to  obtain 
necessary  data. 

6.  Organize  the  data  in  a  form  that  allows  for  analysis. 

7.  Analyze  the  data  to  provide  connections  between  handling  methods  and 
their  consequences  to  a  Lessons  Learned  System. 

8.  Use  the  results  to  suggest  an  appropriate  Handling  characteristic  for  the 
SUPSHIP  Lessons  Learned  System. 


F.  ORGANIZATION  OF  STUDY 

This  thesis  is  a  team  effort.  The  team  consists  of  the  author,  principal  advisor  and 
associate  advisor.  The  author  is  responsible  for  research  and  composition.  The  principal 
advisor  is  responsible  for  guiding  the  author  in  his  research  and  composition  through 
reviews  of  completed  chapters.  The  associate  advisor  is  responsible  for  guidance  in  the 
local  arena  that  is  SUPSHIP  Groton. 

A  thesis  proposal  was  developed  and  approved.  A  product  of  the  thesis  proposal 
is  an  organization  of  the  thesis  by  chapters.  The  chapters  are: 

1.  Chapter  1.  Introduction  This  chapter  describes  the  purpose  of  the 

thesis.  It  also  provides  a  general  background  that  is  helpful  in 
understanding  the  nature  of  the  thesis.  It  includes  the  primary  and 
subsidiary  research  questions. 

2.  Chapter  IT  Literature  Review  This  chapter  provides  a  summary  of  the 
appropriate  research  literature  that  is  relevant  to  the  thesis.  The  existing 
research  literature  provides  a  conceptual  foundation  for  the  thesis. 

3.  Chapter  III.  Methodology  This  chapter  describes  the  approach  used  to 
answer  the  primary  research  question.  It  describes  what  data  is  necessary 
and  how  it  will  be  analyzed. 

4.  Chapter  IV.  Existing  Lessons  Learned  Systems  This  chapter  contains 
the  data  collected  about  existing  Lessons  Learned  Systems.  It  organizes 
the  data  in  a  form  that  can  be  used  in  analysis. 

5.  Chapter  V.  Analysis  This  chapter  analyzes  the  data  and  answers  the 
primary  research  question. 
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6.  Chapter  VI.  Application  to  SUPSHIP  Groton  This  chapter  applies  the 
results  of  the  analysis  to  the  SUPSHIP  Groton  Lessons  Learned  System 

7.  Chapter  VII.  Conclusion  This  chapter  provides  closing  remarks 
concerning  the  primary  research  question,  limitations  of  the  research  and 
recommendations  for  future  research. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

The  purpose  of  this  chapter  is  to  highlight  existing  theoretical  and  research 
information  that  is  relevant  to  this  thesis.  The  existing  theoretical  and  research  literature 
provides  a  conceptual  foundation  for  the  thesis.  It  provides  a  framework  onto  which  the 
work  of  this  thesis  can  be  placed. 

It  also  provides  information  about  Lessons  Learned  System  structures  that  exist. 
This  chapter  provides  the  vocabulary  of  a  Lessons  Learned  System. 

Section  B  provides  general  definitions  relevant  to  a  Lessons  Learned  System. 
Section  C  provides  information  from  Army  Lessons  Learned  System  documents.  It 
discusses  goals  of  a  Lessons  Learned  System  and  provides  the  tasks  that  a  Lessons 
Learned  System  must  perform  in  order  to  achieve  its  goals.  Section  D  provides 
information  on  Department  of  Energy  Lessons  Learned  System  documents.  It  discusses 
goals  and  tasks.  It  also  specifies  the  use  of  root  cause  analysis  for  handling  of  lessons. 
Section  E  provides  analysis  work  done  on  the  operation  of  a  Eessons  Eeamed  System  by 
examining  the  effects  of  different  characteristics.  It  also  formally  defines  the  term 
Handling  as  in  the  handling  of  lessons.  Section  E  provides  information  on  Knowledge 
Management  basics  of  which  a  Eessons  Eeamed  System  is  an  implementation  tool.  It 
includes  numerous  principles  and  suggestions  on  the  handling  of  lessons.  Section  G 
provides  theories  on  Organizational  Eearning.  These  theories  suggest  that  certain 
handling  methods  will  promote  organizational  learning.  Section  H  provides  a  summary. 


B,  DEFINITIONS 

Eearning  is  the  act  of  gaining  knowledge  or  understanding  by  study,  instmction  or 
experience.  12  Included  as  part  of  learning  by  experience  is  learning  by  trial-and-error. 


12  Webster’s  Seventh  New  Collegiate  Dictionary  (1972). 
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In  cognitive  learning  cireles,  discovery  learning  refers  to  trial-and-error  learning.  13 
Learning  is  also  defined  as  a  relatively  permanent  change  in  behavior  that  results  from 
experienoe.i4 

In  general  terms,  a  Lessons  Learned  System  implies  a  system  whose  purpose  is  to 
ereate  some  behavior  as  a  result  of  an  experience  or  a  lesson. 


C.  ARMY  LESSONS  LEARNED  SYSTEM  DOCUMENTS 

The  cognitive  learning  described  above  has  been  taking  place  for  a  long  time. 
Expanding  the  process  from  individual  learning  to  learning  by  a  group  or  organization  is 
also  not  new.  What  is  relatively  new  is  formalizing  the  proeess  and  ereating  a  separate 
system  within  an  organization  to  implement  the  proeess. 

One  of  the  earliest  and  best  known  Lessons  Learned  System  is  the  Center  for 

Army  Lessons  Learned  at  Fort  Leavenworth,  Kansas,  established  in  1985.15 

As  a  Lessons  Learned  System,  the  Center  for  Army  Lessons  Learned  strives  to 
change  Army  behavior  in  a  positive  way. 

Changes  to  behavior  may  result  in  either  stopping  something  we  have  been  doing, 

doing  something  different  from  before,  or  doing  something  new  that  we  have  not 

done  before.  16 

The  goal  is  to  help  soldiers  and  units  perform  their  mission  right  the  first  time, 
regardless  of  the  mission.  17 

The  old  saying  'Live  and  Learn'  must  be  reversed  in  war,  for  there  we  'Learn  and 

Live';  otherwise,  we  die. is 

13  Vander  Zanden,  J.  W.  &  Pace,  A.  J.  (1984),  page  177. 

14  Ibid.,  page  589. 

15  Snider,  K.  F.,  Barrett,  F.  J.,  &  Tenkasi  R.  (2002). 

16  CALL  Handbook  97-13  (1997). 

17  Ibid. 

18  U.S.  War  Department  Pamphlet  No.  20-17,  July  1945. 
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This  is  the  ultimate  goal. 


The  above  guiding  philosophy  of  the  Center  for  Army  Lessons  Learned  supports 
the  literal  interpretation  of  a  Lessons  Learned  System.  A  Lessons  Learned  System  strives 
to  mold  behavior  of  the  organization  or  group  it  serves.  The  new  behavior  is  based  on  a 
learning  experienee  and  it  is  expeeted  that  the  ehange  will  be  positive. 

The  Center  for  Army  Lessons  Learned  uses  a  system  outlined  in  Army  Regulation 
11-  33.  This  regulation  establishes  a  system  for  the  eolleetion,  analysis,  dissemination, 
and  implementation  of  eombat,  training,  and  materiel  testing  experienees  with  assoeiated 
eombat  relevant  lessons  learned  into  Department  of  the  Army  (DA)  doetrine, 
organization,  researeh,  development,  aequisition,  training,  planning,  and  other 
appropriate  aetivities .  1 9 

A  Lessons  Learned  System  has  tasks  that  must  be  aeeomplished  to  aehieve  its 
goals.  Army  Regulation  11-33  has  provided  a  listing  of  these  tasks.  The  first  task  is  to 
eolleet  lessons  or  experienees.  The  seeond  task  is  to  analyze  the  experienees.  During  the 
analysis  phase,  the  raw  data  of  observations  gets  proeessed  into  lessons  through  an 
expanded  interpretation  method  that  ineludes  feedbaek  from  experts  around  the  Army. 20 
The  third  task  is  to  disseminate  the  lessons  of  the  experienees  to  plaees  where  the 
behavior  is  suggested  to  be  ehanged.  The  final  task  is  to  ehange  the  behavior  through 
implementation. 

Army  Regulation  11-33  also  provides  definitions.  It  defines  a  lesson  learned  as 
validated  knowledge  and  experienee  derived  from  observations  and  historieal  study  of 
military  training,  exereises,  and  eombat  operations.21  It  does  not  speoifieally  define 
validate.  In  general,  to  validate  is  to  eonfirm  the  validity,  where  validity  is  the  quality  or 
state  of  being  valid,  where  valid  is  having  a  eonelusion  eorreetly  derived  from 


Army  Regulation  11-33,  Section  1.1  Purpose. 

20  Wizner  A.  (2001),  page  48. 

21  Army  Regulation  11-33,  Glossary. 
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premises. 22  This  is  an  important  function  of  a  Lessons  Learned  System.  This  is 
probably  the  most  intellectually  demanding  function  of  a  Lessons  Learned  System.  It 
requires  seeing  the  general  in  the  specific.  Another  defined  term  is  observation. 
Observation  is  raw  information  from  any  source  that  has  not  been  refined  through 
analysis.  It  can  be  either  positive  or  negative. 23  In  this  definition  there  is  an  expansion 
of  learning  from  mistakes  to  learning  from  mistakes  and  successes. 

The  Army  regulation  glossary  does  not  specifically  define  a  Lessons  Learned 
System  but  does  expand  its  functions  beyond  that  of  collecting,  analyzing  and 
disseminating.  Its  functions  also  include  maintaining  and  managing  an  automated  system 
of  the  experiences  and  lessons.  This  would  support  archiving  for  future  reference,  a 
responsibility  of  the  Commanding  General,  U.S.  Army  Training  and  Doctrine  Command. 
It  is  also  tasked  with  determining  methods  of  dissemination. 


D.  DEPARTMENT  OF  ENERGY  DOCUMENTS 

Another  organization  that  has  a  well-developed  Lessons  Learned  Systems  is  the 
Department  of  Energy.  The  Department  of  Energy  Eessons  Eearned  System  is  guided  by 
a  standard.  The  standard  is  DOE-STD-7501-99,  The  DOE  Corporate  Lessons  Learned 
Program,  December  1999.  The  following  section  summarizes  information  from  DOE  - 
STD-7501-99. 

Under  the  Department  of  Energy,  there  are  many  diverse  projects,  programs  and 
operations.  These  activities  take  place  at  many  different  places.  A  number  of 
Department  of  Energy  rules  and  requirements  require  that  lessons  learned  be  identified, 
evaluated,  shared,  and  incorporated  into  projects,  programs,  or  operations.  These  lessons 
learned  were  mostly  kept  local  and  not  an  integral  part  of  the  Department  of  Energy 
complex.  In  1994  a  Process  Improvement  Team  of  Department  of  Energy  and  contractor 


22  Webster’s  Seventh  New  Collegiate  Dictionary  (1972). 

23  Army  Regulation  11-33,  Glossary. 
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personnel  was  tasked  to  develop  a  technical  standard  to  provide  direction  on  how  to 
develop  Lessons  Learned  Programs.  DOE-STD-7501-99  was  the  result  of  their  work. 

The  purpose  of  a  Lessons  Learned  System  from  this  standard  is  to  share  and  use 
knowledge  derived  from  experience  to:  1)  promote  the  recurrence  of  desirable  outcomes, 
or  2)  preclude  the  recurrence  of  undesirable  outcomes.  The  standard  defines  a  lesson 
learned  as  a  good  work  practice  or  innovative  approach  that  is  captured  and  shared  to 
promote  repeat  application.  A  lesson  learned  may  also  be  an  adverse  work  practice  or 
experience  that  is  captured  and  shared  to  avoid  recurrence.  These  defining  aspects  of  the 
Department  of  Energy  Eessons  Eearned  Program  are  consistent  with  aspects  of  the  Center 
for  Army  Eessons  Learned.  Both  pursue  behavioral  change  that  will  have  positive  effects 
and  both  recognize  that  lessons  or  experiences  that  will  be  used  to  direct  behavior  can  be 
positive  or  negative. 

The  standard  identifies  two  basic  processes.  The  first  is  considered  the 
developmental  process.  This  includes  identification,  documentation,  validation  and 
dissemination.  These  are  actions  that  a  Lessons  Learned  System  would  perform.  The 
second  is  considered  a  utilization  and  incorporation  process.  These  are  actions  that  users 
of  the  system  would  partake  in.  These  include  identifying  applicable  lessons  learned, 
distributing  to  appropriate  personnel,  identification  of  actions  that  will  be  taken  as  a  result 
of  the  lessons  learned,  and  follow  up  actions  to  ensure  the  appropriate  actions  were  taken. 
The  Department  of  Energy  Eessons  Eearned  System  is  meant  to  unite  the  many  local 
Eessons  Eearned  Systems  that  exist  under  its  cognizance.  The  first  process  would  be 
specific  for  the  Headquarters  Eessons  Eearned  System  and  the  second  process  would  be 
for  the  local  Eessons  Eearned  Systems. 

Another  definition  in  the  standard  is  causal  analysis.  A  causal  analysis  is  a  review 
of  an  activity  to  determine  the  root  cause,  to  identify  less  than  adequate  contributing 
systematic  factors,  to  prevent  further  concerns.  This  is  part  of  the  validation  process. 
Many  approaches  are  available  for  identifying  root  causes.  One  of  the  most  effective 
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tools  is  a  cause-and-effect  analysis.  There  are  several  techniques  including  Ishikawa’s 
Fishbone  Diagram  and  Goldratt’s  Thinking  Process. 24 

Another  result  of  the  Process  Improvement  Team  of  Department  of  Energy  and 
contractor  personnel  was  the  formation  of  the  Society  for  Effective  Eessons  Eeamed 
Sharing.  The  Society’s  mission  is  to  promote  the  process  of  identifying,  sharing,  and 
utilizing  lessons  learned  from  experiences  within  the  DOE  complex  and  outside  in  order 
to  improve  the  safety,  efficiency,  and  effectiveness  for  all  Department  work  processes.25 
The  Society  publishes  Fact  Sheets  on  its  website  designed  to  help  Lessons  Learned 
professionals  implement  and  improve  lessons  learned  programs. 26  The  Screening 
Lessons  Learned  for  Site  Applicability  Lact  Sheet  includes  a  flowchart  outlining  a 
decision  process  for  handling  lessons  learned  from  outside  a  local  organization. 

Although  specific  to  local  Department  of  Energy  sites,  it  suggests  a  validation  criteria  for 
lessons  learned  received  from  outside  an  organization.  The  criterion  is  that  lessons 
learned  pertaining  to  similar  activities,  hazards  or  equipment  that  exist  at  the  local  site 
should  be  candidates  for  dissemination  and  action  if  appropriate.  Lessons  that  are  not 
similar  should  be  archived  for  future  reference.  The  archiving  of  lessons  learned, 
including  those  not  disseminated  for  future  reference,  is  a  shared  characteristic  with  the 
Center  for  Army  Lessons  Learned. 


E.  LESSONS  LEARNED  SYSTEM  DECOMPOSITION 

Another  organization  involved  with  Lessons  Learned  Systems  is  the  American 
Association  for  Artificial  Intelligence.  Lounded  in  1979,  the  American  Association  for 
Artificial  Intelligence  is  a  nonprofit  scientific  society  devoted  to  advancing  the  scientific 
understanding  of  the  mechanisms  underlying  thought  and  intelligent  behavior.  The 
American  Association  for  Artificial  Intelligence  activities  include  organizing  and 

24  The  Metrics  Handbook  (1995). 

25  Society  for  Effective  Lessons  Learned  Sharing  Homepage,  http://tis.eh.doe.gov/ll/sells/ 

26  Ibid. 
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sponsoring  conferences,  symposia,  and  workshops. 27  One  workshop  was  on  Intelligent 
Lessons  Learned  Systems. 28  In  the  Call  for  Papers,  a  Lessons  Learned  System  is 
described  as  follows:  Lessons  learned  (LL)  systems  capture  and  store  experiential 
knowledge  for  reuse  in  subsequent  decision-making  tasks. 29 

The  following  seetion  summarizes  information  from  Aha,  D.  W.  (2000)  on 
“intelligent  lessons  learned  systems.” 

A  lecture  was  presented  at  the  Department  of  Energy  Society  for  Effective 
Eessons  Eearned  Sharing  Spring  2000  Meeting.  It  included  observations  on  the  Eessons 
Eearned  process  and  a  characterization  of  Eessons  Eearned  Systems.  Knowledge 
management  is  a  business  movement  that  promotes  knowledge  creation,  sharing  and 
leveraging  within  an  organization  to  maximize  business  results.  In  an  environment  of 
finaneial  constraints  and  loss  of  organizational  knowledge  there  is  a  need  to  develop  a 
culture  of  knowledge  sharing.  This  requires  a  tool  to  capture,  leverage  and  reuse 
knowledge. 

A  lesson  is  a  validated  record  extracted  from  a  (positive  or  failure)  experience 
with  a  previous  decision  process  that  others  in  an  organization  can  reuse  to  reinforce  a 
positive  result  and/or  avoid  a  failure.  A  lesson  learned  is  a  change  resulting  from 
applying  a  lesson  that  significantly  improves  a  targeted  process.  A  Eessons  Eearned 
Process  implements  a  strategy  for  eliciting,  retrieving,  and  reusing  lessons  obtained  from 
experiential  knowledge  to  continually  support  an  organization.  And  a  Eessons  Eearned 
System  is  a  software  system  that  supports  a  Eessons  Learned  Proeess. 

There  is  an  evolution  in  the  definitions  of  lesson  and  lesson  learned.  The  Center 
for  Army  Learned  Lessons  definition  for  lesson  learned  was  a  collection  and  validation  of 
experiences.  The  Department  of  Energy  Eessons  Eearned  Program  included  the  above 
for  lesson  learned  but  also  ineluded  the  sharing  of  the  experienee.  Aha,  D.  W.  (2000) 

27  American  Association  for  Artificial  Intelligence  Homepage 

28  Chaired  by  Dr.  David  W.  Aha  of  the  Naval  Research  Lab  and  Dr.  Rosina  Weber  of  Drexel 
University,  took  place  on  3 1  July  2000  in  Austin,  Texas 

29  Intelligent  Lessons  Learned  Systems  Workshop,  Objectives 
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expanded  the  definition  even  more  to  include  the  above  but  included  the  organization’s 
behavior  being  changed  as  a  result.  The  Aha,  D.  W.  (2000)  definition  of  lesson 
incorporated  the  less  expanded  Army  definition  of  lessons  learned.  The  Aha,  D.  W. 
(2000)  lessons  definition  included  the  positive  and  negative  and  is  consistent  with  the 
Army  and  Department  of  Energy  practice. 

The  definitions  also  provide  a  new  thought  in  the  definition  of  Lessons  Learned 
System.  Aha,  D.  W.  (2000)  moves  the  former  definitions  of  Lessons  Learned  Systems 
into  Lessons  Learned  Process  and  promotes  the  software  (possibly  implied  hardware  also 
to  mean  computer  system  or  information  technology)  system  to  mean  the  Lessons 
Learned  System.  The  use  of  an  automated  system  may  be  more  of  a  requirement  than  a 
luxury  and  the  lecture  redefining  may  be  in  order.  By  the  mid  1990’s,  the  Center  for 
Army  Lessons  Learned  existing  lessons  learned  process  had  been  overwhelmed  with  the 
amount  of  data  that  it  was  collecting;  it  therefore  sought  to  leverage  Information 
Technology  as  a  possible  solution. 

The  automation  of  the  collection  process  proved  invaluable  ...  to  support  all  of 

the  organizations  key  functions.30 

This  implies  that  a  quality  of  a  Lessons  Learned  Systems  (generally  defined)  is 
the  pragmatic  need  to  incorporate  modern  information  technology. 

Aha,  D.  W.  (2000)  also  provides  a  more  formal  characterization  of  Lessons 
Learned  Systems.  The  method  provides  a  characteristic  and  a  qualitative  value  for  the 
characteristic.  Lor  example,  a  characteristic  of  a  Lessons  Learned  System  may  be  size. 
The  qualitative  values  associated  with  size  may  be  large  through  small.  There  were  six 
characteristics.  The  characteristics  are  Contents,  Organizational  Type,  Process  Type, 
Target  Process  Relation,  Dissemination  Type  and  Recommendation. 

The  characteristic  Contents  describes  the  products  of  the  Lessons  Learned 
System.  The  qualitative  values  are  pure  or  hybrid  where  hybrid  may  be  a  collection  of 
lessons,  alerts  or  best  practices.  A  pure  Lessons  Learned  System  would  only  include 


30  Wizner  A.  (2001),  page  19. 
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lessons  while  a  hybrid  Lessons  Learned  System  may  also  inelude  information  that  would 
not  be  classified  as  a  lesson.  The  distinction  between  qualitative  values  is  not  black  and 
white  as  lessons  may  include  positive  experiences  that  some  would  classify  as  best 
practices. 

The  characteristic  Organizational  Type  describes  the  organization  that  the 
Lessons  Learned  System  is  meant  to  serve.  The  qualitative  values  are  adaptable  through 
rigid.  An  adaptable  organization  is  able  to  change  its  processes  and  work  habits  with 
relative  ease.  The  reason  for  this  relative  ease  could  be  that  mechanisms  exist  to 
implement  change  or  that  the  workforce  is  open  to  self-improvement.  A  rigid 
organization  is  one  that  is  not  readily  changed.  The  reasons  for  this  could  be  that  there 
are  many  review  processes  before  processes  could  be  changed  or  that  the  culture  of  the 
organization  is  such  that  change  is  not  easy. 

The  characteristic  Process  Type  describes  the  subject  of  the  lessons.  The 
qualitative  values  are  managerial,  planning  and  technical.  An  example  of  a  managerial 
lesson  would  be  that  a  delivery  from  company  A  is  always  two  weeks  later  than  promised 
so  that  the  order  should  be  placed  two  weeks  earlier  to  receive  the  shipment  “on  time”. 
The  lesson  is  applicable  to  one  person,  the  person  making  the  order.  A  planning  lesson  is 
more  complex  and  involves  many  decision  makers.  An  example  of  this  would  be  a 
political  or  military  campaign.  A  technical  lesson  involves  product  design,  construction, 
test  or  maintenance.  A  technical  lesson  is  also  a  tactic,  technique  and  procedure  for 
operational  military  forces. 

The  characteristic  Target  Process  Relation  describes  how  a  Lessons  Learned 
System  is  integrated  with  the  organization.  The  qualitative  values  are  standalone  and 
embedded.  A  standalone  Lessons  Learned  System  relies  on  user  initiative  to  populate  the 
system  with  lessons.  An  embedded  Lessons  Learned  System  is  integral  with  an 
organization’s  operations.  Operational  procedures  require  that  lessons  be  recorded  and 
entered  into  the  system. 

The  characteristic  Dissemination  Type  describes  how  lessons  are  distributed.  The 
qualitative  values  are  passive  and  active.  A  passive  Lessons  Learned  System  relies  on 
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users  seeking  lessons  from  the  system.  An  active  Lessons  Learned  System  determines 
who  the  appropriate  users  are  and  sends  the  lessons  to  them  for  review  or 
implementation. 

The  characteristic  Recommendation  describes  user’s  authority.  The  qualitative 
values  are  browsable  and  executable.  For  browsable,  the  user  can  only  view 
recommendations.  For  executable,  the  user  can  execute  recommendations. 

The  characteristics  are  not  a  complete  list  of  possible  characteristics  but  provide  a 
possible  framework  for  characterizing  Lessons  Learned  Systems.  The  framework  is  a  set 
of  characteristics  and  qualitative  values  for  those  characteristics. 

Aha,  D.  W.  (2000)  also  provided  a  suggested  list  of  characteristics  that  may  be 
best  for  a  Lessons  Learned  System.  These  suggestions  included  that  a  Lessons  Learned 
System  have  a  Target  Process  Relation  of  embedded.  Another  recommendation  is  that 
the  Lessons  Learned  System  have  a  Dissemination  Type  of  active.  This  was  expanded  to 
include  that  the  information  technology  serve  the  user  and  that  the  user  need  not  be 
proficient  in  the  operation  of  the  information  technology.  The  last  characteristic 
suggested  was  that  the  Recommendation  characteristic  be  executable  or  that  the  users 
have  the  option  to  implement  the  lesson  learned. 

It  is  asserted  by  Aha,  D.  W.  (2000)  that  standalone  (not  embedded),  passive  (not 
active),  browsers  (not  executable)  do  not  promote  knowledge  sharing.  The  reasons  are 
due  to  system  issues  (not  well  integrated  with  other  organizational  processes), 
information  issues  (lessons  not  well  defined)  and  unrealistic  user  assumptions  (users 
know  about  Lessons  Learned  System  and  how  to  use  it,  and  user  can  correctly  interpret 
lesson). 

The  process  of  characterizing  Lessons  Learned  Systems  was  further  refined  in  an 
Acquisition  Review  Quarterly  article.  The  following  section  summarizes  information 
from  Snider,  K.  F.,  Barrett,  F.  J.,  &  Tenkasi  R.  (2002). 

Using  Aha,  D.  W.  (2000)  as  a  basis,  the  characteristics  can  be  grouped  into  one 
of  three  categories.  The  categories  are  Lesson,  Operational  and  Organizational.  It  was 
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suggested  that  the  best  ehoiees  of  eharacteristies  for  an  organization’s  Lessons  Learned 
System  should  be  based  on  the  soeial,  politieal  and  organizational  learning  characteristics 
of  the  organization. 


Table  II- 1  summarizes  the  refinement  with  specific  explanation  following. 


Table  II-l  Lessons  Learned  System  Characteristics 

Group 

Characteristic 

Quantitative  Values 

Lesson 

Content 

pure,  hybrid 

Process  Type 

technical,  administrative,  planning 

Operational 

Access 

open,  closed 

Formality 

formal,  ad  hoc 

Locus 

centralized,  distributed 

Process  Relation 

embedded,  standalone 

Acquisition 

active,  passive 

Handling 

rigorous,  open 

Dissemination 

active,  passive 

Organizational 

Interpretive  Context 

high,  medium,  low 

Type 

adaptable,  rigid 

In  the  paper,  the  Lesson  group  contains  the  characteristics  Content  and  Process 
Type.  These  characteristics  are  the  same  as  presented  in  the  lecture  previously  described. 
The  qualitative  values  are  the  same.  This  group  describes  the  nature  of  the  lessons. 

The  next  group  is  the  Operational  group.  This  describes  how  the  Lessons  Learned 
System  operates.  The  characteristics  are  Access,  Formality,  Locus,  Process  Relation, 
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Acquisition,  Handling  and  Dissemination.  The  Proeess  Relation  eharaeteristic  is  the 
same  as  the  Target  Proeess  Relation  characteristic  of  the  leeture.  The  Dissemination 
eharaeteristie  is  the  same  as  the  Dissemination  Type  of  the  lecture. 

The  eharaeteristie  Aceess  deseribes  the  extent  that  those  outside  the  organization 
may  use  the  organization’s  Lesson  Learned  System.  The  qualitative  values  are  open  and 
elosed.  An  open  Lessons  Learned  System  may  be  used  by  anyone.  Use  is  not  limited  to 
those  in  the  organization.  Closed  means  that  the  Lessons  Learned  System  is  for 
organizational  use  only. 

The  eharaeteristie  Formality  deseribes  the  extent  to  whieh  proeedures  and 
proeesses  are  established.  The  qualitative  values  are  formal  and  ad  hoe.  A  formal 
Lessons  Learned  System  has  doeumented  proeedures  and  proeesses  for  its  operation. 
These  operations  eould  be  for  eolleeting,  validating,  disseminating  and  implementation 
monitoring  to  list  a  few.  An  ad  hoe  Lessons  Learned  System  allows  the  facilitators  to 
decide  any  method  at  any  time. 

The  eharaeteristie  Locus  describes  the  organizational  strueture.  The  qualitative 
values  are  centralized  and  distributed.  A  eentralized  Lessons  Learned  System  relies  on 
one  “offiee”  or  loeation  to  be  the  eenter  for  lessons  learned  and  all  that  goes  with  it.  A 
distributed  Lessons  Learned  System  has  many  loeal  offiees  that  are  performing  lessons 
learned  aetivities  for  a  loeal  part  of  the  organization. 

The  eharaeteristie  Aequisition  deseribes  how  lessons  are  obtained.  The 
qualitative  values  are  aetive  and  passive.  An  aetive  Lessons  Learned  System  seeks  out 
lessons.  This  ean  be  ineorporating  itself  into  the  organization’s  operations  or  seanning 
outside  Lessons  Learned  Systems.  A  passive  Lessons  Learned  System  relies  on 
unsolieited  submission  of  lessons. 

The  qualitative  values  are  rigorous  and  open.  A  rigorous  Lessons  Learned 
System  implies  significant  control  through  some  review  and  approval  proeess.  These 
proeesses  eould  determine  if  the  interpretation  of  the  experienee  is  eorreet,  it  could 
determine  if  the  lessons  learned  is  appropriate  for  dissemination  and  it  eould  also  rewrite 
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the  lesson  to  meet  a  eertain  form  or  writing  standard.  An  open  Lessons  Learned  System 
has  little  or  no  evaluation. 


The  Handling  eharacteristie  is  the  focus  of  this  thesis.  The  Handling 
characteristic  is  examined  for  existing  Lessons  Learned  Systems  and  its  implication  on 
the  effectiveness  of  the  system  with  regard  to  the  tasks  of  a  Lessons  Learned  System. 

The  Organizational  group  contains  the  characteristics  of  the  organization  that  the 
Lessons  Learned  System  is  serving.  The  characteristics  are  Interpretive  Context  and 
Type.  The  Type  characteristic  is  the  same  as  the  Organization  Type  characteristic  of  the 
lecture. 


The  Interpretive  Context  characteristic  refers  to  the  extent  to  which  members  of 
an  organization  share  similar  knowledge,  backgrounds,  and  experiences.  This 
commonality  means  that  communication  is  easy  within  the  organization.  The  qualitative 
values  are  high,  medium  and  low.  An  organization  with  high  Interpretive  Context 
“speaks  the  same  language”  and  is  able  to  communicate  with  one  another  without  an 
“interpreter”. 

The  paper  also  suggested  some  possible  consequences  of  designing  a  Lessons 
Learned  System  with  certain  qualitative  values  of  a  characteristic.  With  regard  to  the 
Handling  characteristic,  a  rigorous  Lessons  Learned  System  may  reduce  participation. 
Processes  of  review,  editing,  validation,  and  approval  may  become  so  burdensome  that 
organizational  members  lose  interest  in  submitting  lessons.  An  open  Lessons  Learned 
System  may  become  populated  with  lessons  that  contain  unsubstantiated  opinions, 
controversial  findings  or  self-serving  claims. 
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F.  KNOWLEDGE  MANAGEMENT 

Like  water,  this  rising  tide  of  data  can  be  viewed  as  an  abundant,  vital  and 
necessary  resource.  With  enough  preparation,  we  should  be  able  to  tap  into  that 
reservoir  —  and  ride  the  wave  —  by  utilizing  new  ways  to  channel  raw  data  into 
meaningful  information.  That  information,  in  turn,  can  then  become  the 
knowledge  that  leads  to  wisdom.  3 1 

This  is  indicative  of  the  mood  that  prevailed  in  the  business  world  in  1995. 

The  development  of  information  and  communication  technology  allows  data  to  be 
abundantly  available.  The  Internet  created  a  new  business  channel. 

The  enhanced  speed  and  capacity  of  communication  has  enabled  the  existence  of 
a  global  market  for  many  industries  and  business  sectors.32 

Consumers  could  access  goods  and  services  from  their  homes.  Manufacturing 
companies  could  search  for  resources  on  a  global  scale. 

Internally,  companies  are  taking  advantage  of  technology  to  retain  data. 

Rapid  changes  in  both  personal  computer  technology  and  electronic 
communications  during  the  past  decade  have  given  us  the  ability  to  create,  gather, 
manipulate,  store,  and  transmit  much  more  data  and  information  than  ever 

before. 33 

With  so  much  data  available  and  with  the  potential  that  goes  with  it,  it  is  not 
surprising  that  management  has  focused  attention  upon  it.  This  focus  is  called  knowledge 
management. 


31  Alberthal,  Les  (1995) 

32  Chase,  R.L.  (1998). 

33  Chase,  R.L.  (1998)  &  Sistla,  M.,  &  Todd,  J.  (1998). 
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Knowledge  management  is  the  systematie  process  of  finding,  selecting, 
organizing,  distilling  and  presenting  information  in  a  way  that  improves  an 
employee's  comprehension  in  a  specific  area  of  interest.  Knowledge  management 
helps  an  organization  to  gain  insight  and  understanding  from  its  own 

experience. 34 

There  is  not  one  universally  accepted  definition  for  knowledge  management  but 
definitions  from  various  sources  are  similar. 

Knowledge  management  involves  the  identification  and  analysis  of  available  and 
required  knowledge,  and  the  subsequent  planning  and  control  of  actions  to 
develop  knowledge  assets  so  as  to  fulfill  organizational  objectives. 35 

Knowledge  management  is  the  process  through  which  organizations  generate 
value  from  their  intellectual  and  knowledge-based  assets. 36 

For  CorpEd.biz,  knowledge  management  is  a  strategy  that  turns  an  organization's 
intellectual  assets  —  both  recorded  information  and  the  talents  of  its  members  — 
into  greater  productivity,  new  value,  and  increased  competitiveness.  It  teaches 
corporations,  from  managers  to  employees,  how  to  produce  and  optimize  skills  as 
a  collective  entity. 37 

Implementing  a  knowledge  management  effort  is  sound  business  strategy. 
Benefits  include  an  increase  in  the  speed  that  an  organization  learns.  Proper  knowledge 
management  will  transform  data  into  knowledge  and  foster  smart  business  decisions.  It 
will  minimize  the  risk  of  making  bad  business  decisions  caused  from  the  use  of  too  much 
or  the  wrong  kind  of  information.38 


34  FAQ 

35  At  At  (2002) 

36  CIO  (2002) 

37  Murray,  P.C. 

38  Peters,  R.F.  (1997). 
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There  are  some  prineiples  assoeiated  with  knowledge  management.  39  Knowledge 
originates  and  resides  in  people’s  minds.  Knowledge  sharing  requires  trust.  Technology 
enables  new  knowledge  behaviors. 

Knowledge  management  started  in  most  companies  as  the  creation  and  use  of 
electronic  repositories. 40 

Knowledge  sharing  must  be  encouraged  and  rewarded.  Management  support  and 
resources  are  essential.  Knowledge  initiatives  should  begin  with  a  pilot  program. 
Quantitative  and  qualitative  measurements  are  needed  to  evaluate  the  initiative. 
Knowledge  is  creative  and  should  be  encouraged  to  develop  in  unexpected  ways. 

There  are  many  terms  or  concepts  that  are  used  within  the  subject  of  knowledge 
management.  One  such  concept  is  that  of  corporate  memory  or  institutional  memory. 

There  is  an  increasing  industrial  interest  in  the  capitalization  of  know-how  of 
(geographically)  dispersed  groups  of  people  in  an  organization.  This  know-how 
may  relate  to  problem  solving  expertise  in  functional  disciplines  (e.g.,  design, 
testing  production),  experiences  of  human  resources,  and  project  experiences  in 
terms  of  project  management  issues  (e.g.  social  and  organizational  aspects  related 
to  the  project  team),  design  technical  issues  (e.g.  design  rationales,  history  of 
solution  space  explored,  concurrent  engineering  techniques),  and  lessons 
learned.4i 

A  sample  of  the  different  activities  that  may  take  place  under  the  umbrella  of 
knowledge  management  is  the  development  of  a  database  of  best  practices  and/or 
lessons  learned  from  failed  projects. 42 

Lessons  learned  and  Lessons  Learned  Systems  are  a  component  of  knowledge 
management. 

Although  the  leading  authorities  do  not  always  specifically  address  Lessons 
Learned  Systems  in  their  writings,  there  is  some  attention  paid  to  data  collection  and 


39  Davenport,  T.H.  &  Prusak,  L.  (1998). 

40  Davenport,  T  (1999). 

41  Workshop  (1996). 

42  Abram  (1997)  and  Broadbent  (1998). 
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information  of  a  pragmatic  nature  that  could  be  applicable  to  the  functioning  of  a  Lessons 
Learned  System. 


Too  much  information  is  almost  as  bad  as  not  enough.  You  have  to  identify 
what’s  relevant,  important  and  effective.43 

This  suggests  that  a  filtering  on  lessons  learned  may  be  appropriate.  This  filtering 
may  take  plaee  in  terms  of  collecting  the  lessons  learned  or  disseminating  the  lessons 
learned. 

Managers  have  come  to  rely  heavily  on  the  computer’s  information.  And  you 
cannot  put  into  the  computer  data  that  you  don’t  have.  Both  executives  and 
students  think  you  tell  the  computer  to  get  the  data,  and  the  computer  gets  it  -no. 
Y ou  have  to  get  it  yourself  44 

This  may  be  applicable  to  the  method  of  lessons  gathering.  The  collection  of 
lessons  learned  of  a  process  should  be  embedded  in  the  implementation  of  the  process  in 
lieu  of  gathering  lessons  learned  after  the  fact  or  in  a  passive  manner. 

Asking  them  to  record  the  lessons  they’ve  learned  during  a  hard  day’s  work,  or  to 
spend  extra  time  searehing  through  an  extensive  repository  before  undertaking  an 
important  task,  is  unlikely  to  meet  with  a  great  deal  of  success.  Instead, 
knowledge  management  has  to  be  “baked  into”  the  job.  It’s  got  to  be  part  of  the 
fabric  of  the  work  to  import  knowledge  when  it’s  needed  and  export  it  to  the  rest 
of  the  organization  when  it’s  created  or  aoquired.45 

In  the  opening  paragraph  of  this  section,  the  evolution  to  wisdom  was  suggested. 
Data  is  eollected  and  transformed  into  information  that  is  transformed  into  knowledge 
that  is  transformed  into  wisdom.  A  better  word  for  wisdom  might  be  behavior  or  actions 
based  on  the  knowledge.  In  the  small  slice  of  knowledge  management  that  is  Lessons 
Learned  Systems,  the  evolution  from  data  to  knowledge  could  be  eonsidered  the 
Handling  characteristic. 


43  Murray,  P.C.  (2000). 

44  Drucker,  P.  (1997). 

45  Davenport,  T.  (1999). 
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Knowledge  management  provides  some  information  on  the  terms  data, 
information,  knowledge  and  behavior .46  Data  are  raw  facts  having  no  meaning  of  their 
own.  Information  is  tangible  representation  of  data  within  a  specific  context.  Knowledge 
is  the  individual  context  on  an  individual’s  role,  learning  behavior  and  experiences. 
Behavior  is  decisions  that  result  in  action. 

We  had  two  decades  which  focused  solely  on  data  processing,  followed  by  two 

decades  focusing  on  information  technology,  and  now  that  has  shifted  to 

knowledge.  There’s  a  clear  difference  between  data,  information,  and  knowledge. 

Information  is  about  taking  data  and  putting  it  into  a  meaningful  pattern. 

Knowledge  is  the  ability  to  use  that  information.47 

This  is  a  reasonable  goal  for  the  Handling  characteristic  of  a  Lessons  Learned 
System.  The  goal  is  to  take  experiences  and  transform  them  into  a  form  that  can  have 
meaning  and  be  of  use  to  the  organization  that  the  Lessons  Learned  System  serves. 

Knowledge  management  also  provides  some  qualities  that  make  information 
valuable.48  The  qualities  are  accuracy  (inspires  confidence),  timeliness  (appropriately 
current),  accessibility  (can  be  readily  located  when  required),  engagement  (capable  of 
making  an  impact  and/or  influencing  a  decision),  application  (relevant  and  useful  within 
the  defined  context)  and  rarity  (possibly  provides  a  hitherto  unknown  or  confidential 
insight).  In  terms  of  applicability  to  a  Lessons  Learned  System,  all  of  these  could  be 
desired  qualities.  In  terms  of  the  Handling  characteristic  of  a  Lessons  Learned  System, 
most  of  these  qualities  could  be  used  to  determine  if  a  lesson  learned  should  be 
disseminated.  The  relevant  qualities  would  be  accuracy,  timeliness,  engagement, 
application  and  rarity. 

Another  view  on  adding  value  to  create  meaningful  information  is  to  customize 
the  data,  categorize  it,  perform  calculations,  make  corrections  to  and  condense  it.  Also  to 


46  Abram,  S.  (1997a). 

47  Blue,  A.  (1998). 

48  Davenport,  T.H.  &  Prusak,  L.  (1997). 
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make  knowledge  more  useful,  it  is  suggested  that  a  comparison  be  provided  and  possible 
consequences  be  determined .49 

Knowledge  management  also  provides  suggestions  for  processing  data  so  that  it 
can  be  absorbed,  applied  and  acted  upon.50  The  first  is  pruning.  Eliminate  the  obsolete, 
the  irrelevant  and  the  inaccurate.  The  second  suggestion  is  adding  context  through 
summary,  analysis,  comparison,  synthesis  and  conclusion.  The  third  suggestion  is 
enhancing  style  through  effective  variation  and  interactivity,  creative  staging  and 
inspirational  dramatization.  The  final  suggestion  is  choosing  the  right  medium  for 
presentation.  There  are  a  number  of  possibilities  for  this  including  an  Intranet,  phone 
calls,  and  E-mails.  The  first  two  suggestions  and  possibly  the  third  may  be  applicable  to 
the  Handling  characteristic  of  a  Eessons  Teamed  System.  The  last  suggestion  would  be 
applicable  to  the  Dissemination  characteristic. 

Knowledge  management  also  provides  some  guidance  for  the  Acquisition 
characteristic  for  a  Eessons  Teamed  System  with  respect  to  the  qualitative  value  of 
active.  It  suggests  the  data  sources  of  the  press  and  other  media,  networking  with  friends, 
associates  and  colleagues,  industry  publications  and  organizational  meetings.  It  also 
suggests  continuing  educational  opportunities,  competitors  or  other  players  in  the  market 
and  any  number  of  other  internal,  external,  formal  and  informal  information  sources. 5 1 
The  reasonableness  of  these  suggestions  would  depend  on  the  nature  of  the  organization 
that  the  Eessons  Teamed  System  serves  but  should  not  be  discarded  outright. 

These  activities  and  others  constitute  a  vital  activity  know  as  ‘environmental 
scanning’,  an  activity  which  no  organization,  regardless  of  its  size,  product  or 
market  position  can  afford  to  ignore. 52 


49  Davenport,  T.H.  &  Prusak,  L.  (1998). 

50  Davenport,  T.H.  &  Prusak,  L.  (1997). 

51  Choo,  C.W.  (1998). 

52  Ibid. 
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G.  ORGANIZATIONAL  LEARNING 

The  key  purpose  of  information:  to  inform  people. 53 

Knowledge  management  goes  beyond  this,  striving  to  ehange  organizational 
behavior  as  a  result  of  this  knowledge.  Changing  organizational  behavior  is  not  a  simple 
task  and  a  braneh  of  knowledge  management  has  eoneentrated  on  faetors  effeeting  the 
ehange  of  organizational  behavior.  That  braneh,  interestingly  enough  older  than 
knowledge  management,  is  organizational  learning. 

Organizational  learning,  for  example,  is  inereasingly  being  drawn  into  the 

knowledge  management  fold.  54 

Knowledge  management  is  about  enhaneing  the  use  of  organizational  knowledge 

through  sound  practices  of  information  management  and  organizational 

learning. 55 

The  transformation  of  data  to  information  to  knowledge  has  no  value  unless  the 
knowledge  is  used  to  guide  organizational  decisions  and  practice.  Sometimes  this 
knowledge  suggests  organizational  decisions  and  practices  that  are  a  significant  change  to 
the  organization,  not  uncommon  in  the  modem  fluid  business  world.  Organizations, 
being  of  large  mass,  so  to  speak,  have  momentum  such  that  change  is  not  always  easy. 
This  has  increased  the  attention  given  to  theories  of  organizational  learning. 

In  1966  Michael  Polanyi  made  the  distinction  between  explicit  knowledge,  which 
can  be  articulated  in  formal  language  and  transmitted  among  individuals,  and  tacit 
knowledge,  personal  knowledge  embedded  in  individual  experience  and  involving  such 
intangible  factors  as  personal  belief,  perspective,  and  values. 56  Within  tacit  knowledge  is 
the  knowledge  that  is  valuable  to  an  organization  and  would  benefit  the  organization  if  it 

53  Davenport,  T.H.  &  Prusak,  L.  (1997). 

54  Davenport,  Thomas  (1999). 

55  Broadbent,  M.  (1998). 

56  Polanyi,  Michael  (1966). 
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were  transferred  to  others  in  the  organization.  This  would  be  for  use  or  future  use. 
However,  taeit  knowledge  is  not  readily  transferable  as  is  explieit  knowledge,  particularly 
between  geographically  separated  regions  of  an  organization. 

Perhaps  the  world’s  most  recognized  authority  on  knowledge  in  the  organization 
is  Ikujiro  Nonaka.  In  his  groundbreaking  book  The  Knowledge-Creating 
Company  (with  co-author  Hirotake  Takeuchi),  he  laid  out  a  model  of  how 
organizational  knowledge  is  created  through  four  major  processes:  socialization, 
extemalization,  combination,  and  internalization.  57 

By  the  creation  of  knowledge  it  is  meant  that  knowledge  is  transferred  from  one 
in  the  organization  to  others;  thus  the  knowledge  exists  in  more  people  and  is  in  a  sense 
created. 


Socialization  is  the  process  where  tacit  knowledge  is  transferred  as  tacit 
knowledge  between  individuals. 58  This  is  accomplished  in  a  “Ba”  (Japanese  signifying 
place,  arena  or  field),  more  precisely  an  “Originating  Ba  where  individuals  can  share 
feelings,  emotions,  experiences  and  mental  models. ”59 

Extemalization  is  the  process  where  tacit  knowledge  is  converted  to  explicit 
knowledge. 60  This  is  accomplished  in  an  Interacting  Ba.  An  example  of  this  is  selecting 
the  people  with  the  right  mix  of  knowledge  and  capabilities  for  a  specific  mission,  like  a 
task  force,  an  urgent  project  team,  or  a  cross-functional  team.6i 

Combination  is  the  process  where  explicit  knowledge  is  transferred  as  explicit 
knowledge  and  absorbed  as  explicit  knowledge. 62 


57  Nonaka  I.  (1997). 

58  Malhotra,  Y.  (1997), 

59  Nonaka,  I.  (1997). 

60  Malhotra,  Y.  (1997), 

61  Nonaka,  I.  (1997). 

62  Malhotra,  Y.  (1997), 
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To  support  the  process  of  knowledge  combination,  Nonaka  suggests  a  Cyber  Ba. 
At  this  point  in  the  process  of  organizational  knowledge  creation,  the  relevant 
knowledge  has  been  captured  and  represented  in  a  way  that  does  not  demand 
face-to-face  human  interaction  to  share.  The  place  for  combination,  therefore,  can 
be  in  the  virtual  world,  using  information  technology  to  transcend  the  limitations 
of  time  and  space. 63 

This  aspect  of  knowledge  creation  is  supported  by  Lessons  Learned  Systems.  In 
these  systems,  explicit  knowledge  is  transferred  between  members  of  an  organization, 
often  by  virtual  world  means. 

Internalization  is  the  process  that  involves  conversion  from  explicit  knowledge  to 
tacit  knowledge. 64  Nonaka  suggests  this  take  place  in  an  Exercising  Ba. 

Here,  the  knowledge  process  being  supported  is  internalization,  in  which  an 
individual  learner  makes  someone  else’s  knowledge  his  or  her  own.  65 

Key  to  the  model  is  Nonaka’ s  assertion  that  none  of  these  processes  is 
individually  sufficient;  all  must  be  present  to  fuel  one  another.  In  fact,  Nonaka  has 
always  said,  it  is  only  when  all  four  processes  interact  that  the  organization  can 
enjoy  a  “spiral”  of  knowledge  creation  -  and  profitable  innovation.66 

Although  a  Lessons  Learned  System  is  most  obviously  part  of  the  Combination 
process,  consideration  should  be  given  in  its  design  and  procedures  for  use  that  support 
the  other  processes.  For  example,  the  Handling  characteristic  of  a  Lessons  Learned 
System  should  prepare  lessons  learned  so  that  not  only  Combination  occurs  but  also 
initiates  Internalization  within  the  receiver  of  the  lesson  learned.  This  would  more  fully 
support  Nonaka’s  model  of  organizational  learning. 

Another  model  in  organizational  learning  is  the  Theory  of  Action.  In  this  theory, 
developed  by  Chris  Argyris  and  Donald  Schon,  members  of  an  organization  have  a  dual 
nature.  One  nature  is  to  make  decisions  and  take  actions  based  on  a  theory-in-use. 


63  Nonaka,  1.(1997). 

64  Malhotra,  Y.  (1997), 

65  Nonaka,  1.  (1997). 

66  Ibid. 
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Theories-in-use  govern  actual  behavior  and  tend  to  be  tacit  structures.  Their 
relation  to  action  is  like  the  relation  of  grammar-in-use  to  speech;  they  contain 
assumptions  about  seif,  others  and  environment  -  these  assumptions  constitute  a 
microcosm  of  science  in  everyday  like. 67 

The  other  nature  is  to  hold  those  theories  espoused  when  asked  to  speak  of  our 
actions  to  others.  This  is  called  espoused  theory.  Argyris  makes  the  case  that 
effectiveness  results  from  developing  congruence  between  theory-in-use  and  espoused 

theory. 68 

This  may  have  some  application  to  a  Lesson  Learned  System  in  the  sense  that 
management  may  state  that  they  want  lessons  learned  on  one  hand  but  require  the 
submitter  to  process  a  large  and  difficult  amount  of  paperwork. 

The  modeling  of  organizational  learning  continues  with  the  concept  of  single-loop 
and  double-loop  learning.  For  Argyris  and  Schon,  learning  involves  the  detection  and 
correction  of  error.  69 

In  single-loop  learning’,  the  detection  and  correction  of  organizational  errors 
permits  the  organization  to  carry  on  its  present  policies  and  achieve  its  current 

objectives. 70 

An  example  of  single-loop  learning  might  be  the  problem  of  maintaining  enough 
coal  in  a  firebox.  A  single-loop  solution  would  be  to  shovel  coal  in  faster. 

Double-loop  learning  occurs  when  error  is  detected  and  corrected  in  ways  that 
involve  modification  of  an  organization’s  underlying  norms,  policies  and  objectives.7i 

A  double-loop  solution  to  the  coal  problem  might  be  to  increase  efficiency  by 
adding  insulation  on  the  firebox  or  reduced  temperature  requirements  or  heat  loads. 


67  Argyris  C.  (2001). 

68  Ibid. 

69  Argyris  C.  (2001). 

70  Argyris,  C.,  and  D.  A.  Schon.  (1978). 

71  Ibid. 
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Double-loop  learning  is  neeessary  if  praetitioners  and  organizations  are  to  make 

informed  deeisions  in  rapidly  changing  and  often  uncertain  contexts. 72 

The  modeling  continues  by  characterizing  organizations  that  would  likely  exhibit 
single-loop  learning  (Model  I)  and  double-loop  learning  (Model  II). 73  There  may  be 
some  consequences  with  regard  to  Lessons  Learned  Systems.  Model  I  members  are 
defensive  and  do  not  wish  to  be  seen  as  incompetent.  This  attitude  would  not  be 
consistent  with  submitting  lessons  learned. 

The  goal  of  this  model  on  organizational  learning  is  to  transform  organizations 
into  Model  II  organizations  such  that  double-loop  learning  will  occur.  There  is  a  strategy 
for  accomplishing  this  and  relies  on  maximum  participation  of  clients,  minimizing  the 
risks  of  candid  participation  and  starting  where  people  want  to  begin.74  One  aspect  that 
may  be  applicable  to  Lessons  Learned  Systems  is  to  implement  group  participation  in  the 
design  of  the  Lessons  Learned  System. 

The  above  represents  a  summary  of  the  major  thinkers  in  the  areas  of  knowledge 
management  and  organizational  learning.  There  is  applicability  to  a  Lessons  Learned 
System  and  particularly  to  the  Handling  characteristic.  The  applicability  is  as  follows. 

A  Lessons  Learned  System  supports  the  Combination  process  of  Nonaka’s 
organizational  learning  model.  It  is  the  transfer  of  explicit  knowledge.  However,  new 
knowledge  begins  with  tacit  knowledge  and  must  be  converted  to  explicit  knowledge  to 
be  transferred,  except  perhaps  in  close  local  environments.  Nonaka  suggests  project  or 
cross  functional  teams  for  this  transfer  while  Drucker  espouses  the  virtues  of  being 
proactive  and  Davenport  suggests  an  embedded  process  for  obtaining  explicit  data.  In 
terms  of  the  Handling  characteristic  it  would  appear  that  handling  should  occur  as  close 
to  the  experience  source  as  possible,  both  in  terms  of  time  and  distance. 

Multiple  authors  suggest  that  the  converted  explicit  knowledge  be  of  appropriate 
form.  Murray  calls  for  the  knowledge  to  be  relevant,  important  and  effective.  Blue  states 

72  Argyris  C.  (2001). 

73  Ibid. 


74  Ibid. 
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that  the  knowledge  should  have  meaning  to  the  user.  Davenport  and  Prusak  advise  that 
the  knowledge  should  be  aecurate,  timely,  applieable  and  eapable  of  influeneing  a 
decision.  Davenport  and  Prusak  also  advise  eliminating  the  obsolete  and  adding  context. 
Nonaka  provides  that  the  knowledge  should  eventually  be  used  to  develop  tacit 
knowledge  in  the  receiver.  For  the  Handling  characteristic,  this  suggests  major 
involvement  from  the  birth  of  the  experience  to  dissemination. 

Argyris  professes  that  users  should  participate  in  the  design.  Although  probably 
not  his  intention,  this  could  be  interpreted  as  the  receivers  participating  in  the  design  of 
the  final  form  (the  product  of  the  Handling  characteristic)  as  it  is  being  developed.  As 
the  lesson  learned  is  being  developed,  before  dissemination,  the  eventual  receiver 
provides  input,  in  an  iterative  fashion  with  the  source  of  the  experience,  to  make  the 
lesson  learned  more  relevant  and  useful. 


H.  SUMMARY 

The  Center  for  Army  Lessons  Learned  has  provided  a  list  of  tasks  that  are  basic  to 
a  Lessons  Learned  System.  They  are:  collect  lessons  or  experiences,  analyze  the 
experience  and  disseminate  the  lessons  to  places  where  the  behavior  is  suggested  to  be 
changed. 

The  Department  of  Energy  has  followed  suit  and  has  identified  basic  requirements 
of  a  Lessons  Learned  Systems.  The  requirements  are  that  lessons  learned  be  identified, 
evaluated,  shared  and  incorporated  into  projects,  programs  and  operations. 
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In  order  to  better  understand  the  operation  of  a  Lessons  Learned  System  and  the 
role  of  environmental  faetors  relating  to  the  organization  it  serves  and  the  nature  of  its 
lessons,  defining  eharaeteristies  of  a  Lessons  Learned  System  have  been  proposed.  These 
are  summarized  in  Table  ILL  Speeifieally  there  is  the  Handling  eharaeteristie. 

Fueled  by  the  development  of  eomputer  networks  and  its  aeeompanying  ability  to 
store  and  transfer  information,  a  study  of  information  transfer  has  developed  entitled 
Knowledge  Management.  Under  the  subjeet  of  Knowledge  Management  resides  Lessons 
Learned  Systems.  Knowledge  Management  has  provided  some  basie  prineiples  that 
relate  to  the  handling  of  lessons  learned.  These  are  ineluded  in  Table  II-2. 

A  basie  goal  of  a  Lessons  Learned  System  is  the  implementation  of  lessons  sueh 
that  behavior  is  ehanged  in  a  positive  way.  Changing  behavior  on  an  organizational  seale 
is  ineluded  under  the  subjeet  of  Organizational  Learning.  Organizational  Learning 
theories  suggest  eertain  aetions  relating  to  the  handling  of  lessons  learned.  These  are  also 
ineluded  in  Table  II-2. 


34 


Table  II-2  Literature  Principles  that  Relate  to  Handling  of  Lessons 

Author 

Statement 

Center  for  Army  Lessons  Learned 

validate  (conclusion  correctly  derived 
from  premises)  the  lesson 

Department  of  Energy 

perform  causal  analysis  to  determine  root 
cause  of  lesson 

Murray,  Phillip 

filter  lessons  for  relevancy 

Drucker,  Peter 

information,  you  have  to  get  it  yourself 

Davenport,  Tom 

performing  work  and  asked  to  record 
lessons  while  doing  so  is  unlikely  to  be 
successful 

Blue,  A. 

transform  information  to  knowledge 

Davenport,  Tom  &  Prusak,  L. 

customize  data 

Nonaka,  1. 

lessons  should  initiate  the  Internalization 
process 

Argyris,  Chris 

management  support  and  involvement 

Argyris,  Chris 

users  participate  in  process/design 

Argyris,  Chris  &  Schon,  D.  A. 

root  cause  analysis  (double  loop 
learning) 

The  above  provides  the  eoneeptual  framework  by  whieh  to  evaluate  the  primary 
research  question:  How  may  a  Lessons  Learned  System  be  characterized  according  to  the 
handling  of  lessons  processing,  and  how  may  such  a  characterization  be  applied  to 
Lessons  Learned  System  design  or  architecture? 
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III.  METHODOLOGY 


A.  INTRODUCTION 

The  purpose  of  this  ehapter  is  to  describe  the  approach  used  to  answer  the  primary 
research  question:  How  may  a  Lessons  Learned  System  be  characterized  according  to  the 
handling  of  lessons  processing,  and  how  may  such  a  characterization  be  applied  to 
Lessons  Learned  System  design  or  architecture? 

This  section  describes  the  overall  methodology  while  the  remaining  sections 
provide  more  detail  on  the  specific  tasks  of  the  methodology. 

Chapter  II  has  provided  a  framework  for  characterizing  Lessons  Learned  Systems. 
A  Handling  characteristic  has  been  defined. 

Unfortunately,  the  literature  review  reveals  there  is  some  cloudiness  concerning 
what  a  lesson  is.  The  first  task  of  the  analysis  will  be  to  more  precisely  define  lesson  and 
therefore  allow  a  starting  point,  in  terms  of  time,  when  actions  of  a  Lessons  Learned 
System  would  be  considered  actions  of  Handling. 

The  qualitative  values  associated  with  the  Handling  characteristic  are  rigorous  or 
open.  However,  this  is  not  sufficient  to  characterize  a  Lessons  Learned  System  according 
to  the  handling  of  lessons  processing  to  the  extent  that  the  specifics  of  the  rigorous  value 
can  be  various.  Therefore  the  rigorous  value  will  be  expanded  into  a  rigorous  set.  The 
rigorous  set  will  include  the  handling  possibilities. 

As  it  is  possible  that  a  Lessons  Learned  System  will  not  incorporate  all  the 
possible  rigorous  methods,  the  rigorous  set  will  include  the  ability  to  specify  the 
existence  of  the  method  or  its  omission.  This  property  allows  the  rigorous  set  to  be 
renamed  the  rigorous  variable  set.  As  rigorous  methods  may  be  variable,  the  omission  of 
all  would  indicate  a  qualitative  value  of  open.  The  rigorous  variable  set  is  more 
appropriately  named  the  Handling  Variable  Set. 

The  existence  of  a  Handling  Variable  Set  answers  the  first  part  of  the  primary 
research  question:  How  may  a  Lessons  Learned  System  be  characterized  according  to  the 
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handling  of  lessons  processing?,  in  a  more  precise  and  encompassing  manner.  The 
specifics  of  the  Handling  Variable  Set  will  be  developed  in  Section  E  of  this  chapter  and 
Chapter  V. 

Chapter  II  has  also  provided  a  list  of  tasks  that  are  basic  to  a  Lessons  Learned 
System.  They  are  collect  lessons  or  experiences,  analyze  the  experience  or  lessons  to 
obtain  knowledge  and  disseminate  the  knowledge  to  places  where  the  behavior  is 
suggested  to  be  changed.  These  tasks  can  be  reworded  as  receiving  lessons,  developing 
quality  of  lessons  and  disseminating  to  insure  implementation. 

On  the  subject  of  research  methods. 

We  start  by  carefully  considering  what  it  is  we  already  know  and  thus  what  it  is 

we  need  to  find  out  about  through  research.75 

Known  are  the  tasks  of  a  Lessons  Learned  system.  What  is  not  known  is  how  the 
Handling  Variable  Set  affects  these  tasks. 

Obtaining  this  knowledge  will  allow  the  second  part  of  the  primary  research 
question  to  be  answered.  The  second  part  of  the  research  question  is:  and  how  may  such 
a  characterization  be  applied  to  Lessons  Learned  System  design  or  architecture?  The 
method  of  determining  this  knowledge,  the  knowledge  being  the  cause  and  effect 
relationship,  is  detailed  in  Section  L. 

The  following  sections  provide  the  specific  tasks  of  the  methodology. 


75  Harvey,  D.  (2002). 
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B.  REQUIRED  DATA 


The  Handling  characteristic  of  Lessons  Learned  Systems  is  not  universal  and  is 
unique  to  each  Lessons  Learned  System.  In  order  to  collect  methods  of  handling  and 
their  effect,  Lessons  Learned  Systems  must  be  identified.  Once  these  systems  have  been 
identified,  a  method  of  contact  must  be  determined. 

The  extent  of  what  a  Lessons  Learned  System  considers  handling  may  also  vary. 
Therefore  information  on  the  entire  process  is  necessary  to  determine  what  potentially 
could  be  considered  the  Handling  characteristic.  This  will  provide  development  of  the 
Handling  Variable  Set. 

In  order  to  determine  how  the  Handling  Variable  Set  may  be  applied  to  Lessons 
Learned  System  design  or  architecture,  a  number  of  pieces  of  information  will  be 
required.  These  include  the  purpose  or  goals  of  the  Lesson  Learned  System,  the  present 
concerns,  the  consequential  experiences  of  the  utilized  Handling  characteristic  and  all 
other  characteristics  defined  in  Table  ILL 

To  summarize,  every  aspect  of  the  Lessons  Learned  System  and  the  organization 
it  serves  could  have  some  relevancy;  therefore  as  much  information  as  possible  on  a 
particular  Lessons  Learned  System  and  its  organization  will  be  obtained  along  with 
experiences  from  its  operation. 


C.  DATA  COLLECTION 

The  identification  of  potential  Lessons  Learned  Systems  was  the  starting  point. 
The  Internet  provided  a  listing  of  existing  Lessons  Learned  Systems. 76  The  Web  pages 
for  these  organizations  were  reviewed.  An  attempt  was  made  to  contact  each 
organization  about  its  Lessons  Learned  System.  Most  provided  an  e-mail  address.  In 
these  cases,  an  e-mail  was  sent  requesting  that  an  attached  questionnaire  to  the  e-mail  be 


76  Lessons  Learned  Links,  http://www.aie.nrl.navv.mil/~aha/lessons/ 
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filled  out  and  returned.  In  eases  where  a  phone  number  was  also  listed,  an  attempt  was 
made  to  personally  make  eontact  with  a  Lessons  Learned  Systems  person.  Cooperation 
with  the  Lessons  Learned  System  was  sueeessful  in  cases  where  personal  contact  was 
made. 


The  questionnaire  was  designed  with  questions  related  to  the  primary  research 
question.  Being  that  a  project,  be  it  design  or  a  paper,  is  iterative  in  nature,  the 
questionnaire  was  not  ideal.  The  implementation  of  the  analysis  revealed  the 
questionnaire’s  shortcomings.  To  counter  this  deficiency,  e-mails  and  phone  calls  were 
used  to  collect  additional  data. 

The  questions  on  the  questionnaire  are  listed  below  with  the  original  intended 
purpose  of  the  question  in  parenthesis.  The  information  obtained  sometimes  provided  an 
insight  that  was  not  originally  envisioned. 

The  questions  on  the  questionnaire  were: 

1 .  What  is  the  purpose  of  the  Lessons  Learned  System?  (This  was  asked  to 
find  out  about  the  Lessons  Learned  System.  Answers  to  this  question 
might  answer  questions  such  its  function  and  overall  integration  into  the 
organization.) 

2.  How  are  lessons  obtained?  (This  was  asked  to  collect  information  on 
events  that  lead  up  to  the  handling  of  the  lessons  and  for  general 
information.  This  provided  information  on  the  subject  of  the  lessons.) 

3.  What  degree  of  formality  is  used,  if  any,  in  validating  the  lessons  learned? 
(Validating  of  the  lessons  was  considered  the  main  activity  of  handling 
and  so  this  question  was  asked  to  obtain  information  on  the  nature  of 
handling.) 

4.  Why  was  this  degree  of  formality  used?  (This  was  asked  to  probe  if  there 
was  a  relationship  between  a  very  formal  processing  and  the  importance  of 
the  lesson.) 

5.  What  is  the  validation  process?  (This  was  asked  to  obtain  an 
understanding  of  the  mechanics  of  the  handling  process.) 

6.  Has  the  Lessons  Learned  System  been  successful?  Based  on  what 
evidence?  (This  was  asked  to  provide  a  check  that  the  method  of  handling 
based  on  the  purpose  of  the  Lessons  Learned  System  was  appropriate  as 
indicated  by  success.  The  second  part  of  the  question  was  to  reduce 
subjectivity.) 
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7.  What  are  the  consequenees  of  disseminating  an  erroneous  or  misleading 
lessons  learned?  (This  was  to  probe  again  if  there  was  a  relationship 
between  a  very  formal  proeessing  and  the  importance  of  the  lesson) 

8.  Is  there  a  disclaimer  associated  with  the  Lessons  Learned  System?  (This 
was  to  gauge  the  confidence  of  the  Lessons  Learned  System  in  their 
handling  methods  in  terms  of  disseminating  accurate  information.) 

9.  Is  there  a  single  person  responsible  for  the  accuracy  of  the  Lessons 
Learned  System?  (This  was  to  identify  an  additional  contact  person  and 
gauge  accuracy  of  the  lessons.  If  there  is  not  one  person  responsible  then 
there  is  no  one  really  responsible. 7V) 

Some  organizations  responded  to  the  questionnaire.  In  all  cases  it  should  be 
understood  that  the  answers  to  the  questions  are  not  necessarily  the  official  answer  or 
position  of  the  organization. 

Another  method  of  obtaining  information  on  existing  Lessons  Learned  Systems 
was  by  direct  contact  where  direct  contact  was  feasible.  Again,  information  obtained 
through  direct  contact  is  not  to  be  considered  the  official  policy  of  the  organization. 


D,  DATA  ORGANIZATION 

The  data  received  from  the  questionnaire  supplemented  by  e-mails  and  telephone 
calls  was  used  for  a  general  write-up  of  each  Lessons  Learned  System  (see  Chapter  IV). 
In  order  to  ease  abstraction  of  relevant  data  from  the  write-ups,  the  data  was  organized 
into  tables. 


E.  HANDLING  VARIABLE  SET 

The  development  of  the  Handling  Variable  Set  was  based  on  empirical  data. 
From  the  data  on  existing  Lessons  Learned  Systems,  the  handling  methods  of  each 
Lessons  Learned  System  were  listed  in  a  table.  From  that  table,  common  methods  were 
combined  and  ordered  in  the  logical  time  progression  of  actions  of  a  Lessons  Learned 
System.  The  order  was:  receiving  lessons,  analyzing  lessons  and  finally  actions  related  to 


77  Admiral  Rickover  on  requiring  signatures  on  doeuments 
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dissemination.  This  list,  representing  the  domain  of  handling  methods,  was  used  to  ereate 
the  Handling  Variable  Set. 

This  list,  placed  in  a  table,  was  transformed  into  a  number.  The  number  was  a 
collection  of  binomial  numbers.  Each  placeholder  of  the  number  represented  a  handling 
method.  That  is,  the  ones  place  represented  a  handling  method,  the  tens  place 
represented  another  handling  method,  the  100s  place  another,  etc.  A  one  in  the 
placeholder  represented  that  the  handling  method  was  employed,  a  zero  meant  that  it  was 
not  used. 

As  an  illustration  of  the  method,  consider  a  sample  of  three  Lessons  Learned 
Systems.  It  was  identified,  through  a  questionnaire,  that  Lesson  Learned  System  1 
employed  handling  method  A,  Lessons  Learned  System  2  employed  handling  method  B 
and  Lessons  Learned  System  3  employed  handling  methods  A  and  B. 

The  domain  of  handling  methods  would  then  be  A  and  B.  Since  there  are  two,  a 
two  digit  number  would  be  used;  A  being  represented  by  the  tens  place  and  B  being 
represented  by  the  ones  place.  A  one  is  used  to  represent  existence  of  the  method  and  a 
zero  is  used  to  indicate  omission. 

Lessons  Learned  System  1  would  then  have  a  Handling  Variable  Set  of  10 
because  it  employs  handling  method  A  (indicated  by  a  one  in  the  tens  place)  and  does  not 
employ  handling  method  B  (indicated  by  a  zero  in  the  ones  place).  Lessons  Learned 
System  2  would  have  a  Handling  Variable  Set  of  01  because  it  employs  handling  method 
B  (indicated  by  1  in  the  ones  place)  but  does  not  employ  handling  method  A  (indicated 
by  0  in  the  tens  place).  Lessons  Learned  System  3  would  have  a  Handling  Variable  Set 
of  1 1  because  it  employs  both  handling  methods  A  and  B  (indicated  by  1  in  the  tens  place 
and  a  1  in  the  ones  place). 

This  coding  allowed  bulky  qualitative  data  to  be  represented  in  a  much-condensed 
form.  It  provided  a  method  to  accurately  characterize  or  describe  a  handling  method  for 
any  existing  Lessons  Learned  System,  at  least  within  the  database  of  Chapter  IV  and 
allowed  quicker  comparisons. 
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F. 


APPLICATION  TO  LESSONS  LEARNED  SYSTEM  DESIGN 


The  approach  taken  was  to  determine  what  effects  existing  Handling  Variable  Set 
digit  values  had  on  the  tasks  of  a  Lessons  Learned  System.  These  tasks  were  the  main 
tasks  identified  in  Chapter  IT  These  tasks  were  receiving  lessons,  quality  of  lessons 
learned  disseminated  and  dissemination  of  lessons  learned  where  dissemination  included 
implementation. 

This  is  a  qualitative,  empirical  analysis.  The  tool  or  method  used  to  provide  the 
framework  for  this  analysis  is  the  influence  diagram. 

The  influence  diagrams  are  used  both  as  a  means  for  communicating  the  model 
between  various  categories  of  "experts",  further  as  an  aid  in  accident  analyses, 
and  also  in  the  qualitative  evaluation.  78 


1.  The  Influence  Diagram 

An  influence  diagram  is  a  graphical  representation  of  the  influences  on  some 
objective.  Usually  the  objective  is  to  maximize  or  minimize  some  attribute.  The 
influences  are  identified,  distinguished  by  symbol,  as  those  that  are  controllable  and  those 
that  are  not. 


78  Hokstad,  P. 
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Table  III-l  identifies  the  symbols  used  and  their  meaning. 


Table  III-l  Influence  Diagram  Symbols 

Name 

Symbol 

Meaning 

Rectangle 

A  decision,  a  variable  that 
the  decision  maker  has  the 
power  to  control 

Oval 

CO 

A  variable  that  is  not 
controllable  by  the  decision 
maker,  an  environment  or 
chance  condition 

Hexagon 

o 

An  objective  variable, 
criterion  that  is  to  be 
maximized  or  minimized 

Arrow 

- ► 

Denotes  influence 

2,  Influence  Diagram  Variables 

An  influence  diagram  has  one  objeetive  variable.  The  objeetive  variables  that  are 
of  interest  to  Lessons  Learned  System  design  are  receiving  lessons  (maximize),  the 
quality  of  lessons  (maximize)  and  the  implementation  of  lessons  (maximize).  There  are 
three;  therefore  there  will  be  three  Influence  Diagrams. 

The  decision  variables  are  the  choices  that  can  be  made  that  effect  the  objective 
variables.  For  our  purposes,  the  decision  variable  of  interest  is  the  Handling  Variable 
Set,  as  this  relates  to  the  second  part  of  the  primary  research  question.  This  is  not  to 
exclude  other  non-Handling  characteristics  that  may  effect  the  second  part  of  the  primary 
research  question. 

The  environment  or  chance  variables  are  those  variables  that  the  Lessons  Learned 
System  designer  has  no  control  over. 
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It  is  not  necessary  to  provide  a  complete,  all  encompassing  Influence  Diagram. 
There  is  only  a  need  for  an  Influence  Diagram  that  contributes  to  answering  the  second 
part  of  the  primary  research  question. 


3,  Determining  the  Influence  Diagram 

The  determination  of  the  influence  diagram  will  be  based  on  empirical  data  from 
Chapter  IV. 

The  main  method  will  be  the  use  of  commonality.  An  example  would  be  all 
Lessons  Learned  Systems  that  experience  success  in  one  objective  employ  a  certain 
handling  method.  Those  that  do  not  employ  the  handling  method  do  not  experience 
success.  That  handling  method  is  then  shown  as  a  decision  variable  on  the  Influence 
Diagram. 

The  proposed  theoretical  suggestions  for  the  Handling  characteristic  can  then  be 
used  to  support  the  findings  and  vice  versa. 


4,  Example  of  Determining  the  Influence  Diagram 

In  Section  E,  the  method  of  determining  the  Handling  Variable  Set  was 
illustrated.  Lessons  Learned  System  1  had  a  Handling  Variable  Set  of  10,  Lessons 
Learned  System  2  had  a  Handling  Variable  Set  of  01  and  Lessons  Learned  System  3  had 
a  Handling  Variable  Set  of  1 1 . 

Through  use  of  a  questionnaire  and  interviews  it  was  revealed  that  Lessons 
Learned  System  2  has  a  concern  for  receiving  lessons  while  Lessons  Learned  Systems  1 
and  3  did  not  have  this  concern.  Comparing  the  Handling  Variable  Sets  of  the  three,  it 
can  be  seen  that  there  is  evidence  that  the  tens  place  effects  receiving  lessons.  A  one  in 
the  tens  place  and  there  is  no  concern,  a  zero  and  there  is  a  concern.  From  Section  E,  a 
one  in  the  tens  place  represents  implementation  of  handling  method  A. 
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An  influence  diagram  can  then  be  constructed  with  the  Handling  Variable  Set  of 
lx  (where  x  could  be  one  or  zero)  as  a  decision  variable  (rectangle)  and  receiving  lessons 
as  an  objective  (hexagon).  This  cause  and  effect  can  then  be  substantiated  by  the 
literature  review.  The  literature  review  may  suggest  that  handling  method  A  will 
promote  receiving  lessons. 


5,  Use  in  Lessons  Learned  System  Design 

Influence  Diagrams  can  be  used  to  determine  if  a  proposed  design  or  architecture, 
in  terms  of  lessons  handling,  will  be  successful  in  the  three  main  tasks  that  a  Lessons 
Learned  system  must  perform  to  be  successful.  The  Handling  Variable  Set  for  the 
Lessons  Learned  system  is  determined  and  compared  to  the  information  listed  in  the 
Influence  diagrams.  An  omission  of  a  handling  method  may  indicate  potential  problems 
in  one  of  the  tasks. 


G.  APPLICATION  TO  THE  SUPSHIP  GROTON  LESSONS  LEARNED 

SYSTEM 

The  method  of  characterizing  the  handling  of  lessons  in  a  Lessons  Learned 
System  by  the  Handling  Variable  Set  and  the  use  of  the  Influence  Diagrams  to  Lessons 
Learned  System  Design  can  be  applied  to  the  new,  developing  SUPSHIP  Groton  Lessons 
Learned  System. 

The  approach  will  be  to  examine  the  present  Handling  Variable  Set  of  the 
SUPSHIP  Groton  Lessons  Learned  System  against  the  Influence  Diagrams  to  predict 
performance  in  the  areas  of  receiving  lessons,  quality  of  lessons  learned  disseminated  and 
implementation  of  the  lessons  learned  by  SUPSHIP  Groton.  Where  sub-performance  is 
predicted,  suggested  changes  to  the  method  of  handling  lessons,  which  will  change  the 
Handling  Variable  Set  accordingly  to  remove  the  sub  performance  prediction,  will  be 
made. 
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H.  CONCLUSION 


This  chapter  provided  the  methodology  to  answer  the  primary  researeh  question.  It  is  an 
empirieal,  qualitative  method  based  on  the  data  of  Chapter  IV. 
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IV.  EXISTING  LESSONS  LEARNED  SYSTEMS 


A,  INTRODUCTION 

This  chapter  contains  the  data  collected  about  existing  Lessons  Learned  Systems. 
It  also  organizes  the  data  in  a  form  that  supports  the  analysis  methodology  of  Chapter  III 
and  is  implemented  in  Chapter  V. 

To  obtain  information  about  Lessons  Learned  Systems,  the  Internet  was  the 
primary  resource  used.  It  provided  a  listing  of  Lessons  Learned  Systems. 79  From  the 
list,  organizations  employing  Lessons  Learned  Systems  could  be  accessed.  As  detailed  in 
Chapter  III,  information  was  then  obtained  through  a  questionnaire  and/or  an  interview. 

Section  B  contains  write-ups  on  the  existing  Lessons  Learned  Systems  based  on  a 
questionnaire  and  e-mail  and  telephone  follow-up.  Section  C  organizes  the  data  in  tables. 


B.  EXISTING  LESSONS  LEARNED  SYSTEMS 

The  phrase  existing  Lessons  Learned  System  represents  a  sample  of  Lessons 
Learned  System  and  is  not  meant  to  represent  all  existing  Lessons  Learned  Systems. 

Some  organizations  responded  to  the  questionnaire.  In  all  cases  it  should  be 
understood  that  the  answers  to  the  questions  are  not  necessarily  the  official  answer  or 
position  of  the  organization  answering. 

The  organizations  that  responded  were; 

1 .  Canadian  Army  Lessons  Learned  Centre 

2.  The  United  Nations  Peacekeeping  Best  Practices  Unit  (formerly  The 
Lessons  Learned  Unit  of  the  Department  of  Peacekeeping  Operations) 

3.  American  Industrial  Hygiene  Association  (AIHA)  Health  &  Safety 
Committee 

4.  U.S.  Army  Center  for  Engineer  Lessons  Learned 

5.  Army  Medical  Department  Lessons  Learned 

79  Lessons  Learned  Links,  http://www.aie.nrl.navv.mil/~aha/lessons/ 


49 


6.  Coast  Guard  -  Standard  After  Action  Information  and  Lessons  Learned 
System 

7.  Best  Manufacturing  Practices  Program 

8.  Department  of  Energy  Project  Hanford  Lessons  Learned  System 

9.  BNFL  Incorporated  Lessons  Learned  System 

10.  Department  of  Energy  Headquarters  Eessons  Eearned  System 

11.  Department  of  Energy  Office  of  Environmental  Management  Eessons 
Eearned  Program 

12.  Federal  Transit  Administration  Eessons  Eearned  System 

13.  International  Space  Station  Eessons  Eearned  Database 

14.  Mine  Action  Information  Center  Eessons  Eearned  Database 

15.  Electric  Boat  Corporate  Eessons  Eearned  Database 

1.  Canadian  Army  Lessons  Learned  Centreso 

These  responses  are  provided  from  my  experience  working  in  the  Canadian  Army 
Lessons  Learned  Centre,  as  well  as  the  experience  of  others  in  the  section, 
however  the  responses  are  not  necessarily  Canadian  Army  policy  or  direction. 
My  comments  are  provided  to  you  for  background  material  on  lessons  learned 
organizations. 81 

Within  the  Canada  National  Defense  exist  the  Land  Force  Doctrine  and  Training 
System  Formation.  At  their  headquarters  in  Kingston,  Ontario  is  the  strategic  staff 
entitled  Lessons  Learned.  This  is  the  Canadian  Army  Lessons  Learned  Centre. 

The  purpose  of  the  Lessons  Learned  System  is  to  collect  and  analyze  Canadian 
and  Allied  operational  and  training  experiences  for  dissemination  as  Lessons  Learned,82 
with  a  view  to  improving  the  overall  operational  capability  of  the  Army.  This  includes 
efficiency  of  present  operations  and  expanded  operational  capability  through  improved 

80  Twohey,  Major.  J.  Mark,  Canadian  Army  Lessons  Learned  Centre  point  of  contact, (personal 
communication,  e-mail  questionnaire,  January  22,  2002) 

81  Ibid. 

82  Canadian  Army  Lessons  Learned  Centre  Web  Page 


50 


technology  such  as  bullet  proof  vests  and  warmer  deep  winter  boots. 83  The  three  key 
activities  are  collect,  analyze  and  disseminate.  The  ultimate  purpose  of  the  Lessons 
Learned  Centre  is  to  help  promote  positive  change. 

Lessons  are  obtained  in  several  ways.  Two  primary  methods  are  by  (1) 
documented  reports  on  training  or  operations  and  (2)  visits  to  units  on  training  and 
operation.  The  documented  reports  contain  questions  that  are  answered  by  the  unit 
involved  in  the  operation  or  training.  The  questions  are  general  or  broad  so  that  the  same 
questions  can  be  used  for  different  training  or  operation.  This  creates  standard  reports 
although  there  are  different  reports  for  training  and  operations.  The  training  reports  are 
shorter  and  completed  after  the  exercise. 

The  operational  reports  are  longer  and  are  completed  in  two  parts.  Operations  are 
divided  into  phases.  Phase  1  is  Warning,  Phase  2  is  Preparation,  Phase  3  is  Deployment, 
Phase  4  is  Employment  and  Phase  5  is  Redeployment.  The  first  part  of  the  operational 
report  is  for  Phases  1,  2  &  3  and  is  submitted  6  weeks  after  deployment  and  covers 
activities  that  took  place  in  Canada  during  these  phases.  The  second  part  is  to  be 
completed  6  months  at  the  end  of  the  tour.  The  reason  for  their  being  two  parts  is  that 
lessons  captured  in  the  early  phases  can  be  passed  on  to  others  without  waiting  for  an 
operation  to  conclude  and  memories  of  the  early  phases  are  still  fresh  in  the  minds  of  the 
people  completing  the  form. 

Reports  are  sent  up  through  the  chain  of  command  for  comment,  with  information 
copies  sent  to  the  Lessons  Learned  Centre  to  keep  them  informed  of  the  flow  of 
information.  One  value  of  the  standard  form  is  that  an  issue  can  be  tracked  with  respect 
to  the  forms  that  are  received  from  different  sources.  The  forms  also  include  opportunity 
for  the  units  to  add  miscellaneous  comments. 

The  Lessons  Learned  Centre  also  visits  exercises  in  Canada  as  well  as  deployed 
operations  in  order  to  stay  up  to  date  with  what  is  happening  in  the  field,  and  also  to  have 
a  better  perspective  when  reading  the  reports  after  the  fact. 


83  Young,  Major  J.,  Canadian  Army  Lessons  Learned  Centre  point  of  contact,  (personal 
communication,  telephone,  July  2,  2002) 


51 


The  Canada  Lessons  Learned  Centre  defines  four  key  events  that  apply  to  the 
lessons  learned  proeess: 

1.  Observation  -  An  observation  is  a  eomment  about  an  experience  that 
occurred  during  an  operation,  training  event  or  other  activity. 
Observations  provide  the  data  upon  which  analysis  is  subsequently 
conducted. 

2.  Issue  -  An  issue  is  a  topic  that  develops  from  one  or  more  related 
observations  or  recurring  observations. 

3.  Lesson  -  The  knowledge  that  is  generated  from  the  analysis  of  an 
observation  to  determine  the  underlying  causes,  the  implications  and 
which  can  subsequently  be  used  to  plan  effective  action. 

4.  Lesson  Learned  -  A  lesson  learned  is  a  lesson  that,  when  assimilated, 
resulted  in  a  tangible  change  in  attitude,  capability,  behavior  or  process. 


The  reports  then  act  as  observations.  The  Canada  Lessons  Learned  Centre  then 
analyzes  the  observations  to  determine  what  actions  are  necessary.  These  actions  could 
effect  doctrine,  training,  acquisition  of  equipment  and  so  forth.  The  Lessons  Learned 
Centre  does  not  implement  these  changes  but  advises  authority  as  to  reporting  and 
change/implementation.  The  reports  are  the  main  vehicle  to  obtain  lessons. 

In  terms  of  the  formality  used,  this  is  viewed  by  the  Lessons  Learned  Centre  as 
follow  up  to  direction  on  change  defined  as  lesson  learned.  The  Lessons  Learned  Centre 
states  that  this  can  be  accomplished  by  several  different  functions,  most  importantly  the 
chain  of  command. 

What  is  equally  important  to  note,  is  that  just  because  direction  was  passed  on  to 

implement  change,  it  doesn’t  necessarily  translate  into  action  at  lower  levels, 

which  brings  me  back  to  the  importance  of  the  chain  of  command  in  the 

process. 84 

The  Lessons  Learned  Centre  is  not  a  command  organization.  The  implementation 
of  lessons  learned  is  the  present  concern  of  the  Lessons  Learned  Centre. 85 

84  Twohey,  Major.  J.  Mark,  Canadian  Army  Lessons  Learned  Centre  point  of  contact,  (personal 
communication,  e-mail  questionnaire,  January  22,  2002) 

85  Young,  Major  J.,  Canadian  Army  Lessons  Learned  Centre  point  of  contact,  (personal 
communication,  telephone,  July  2,  2002) 
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Another  point  brought  out  with  regard  to  the  degree  of  formality  is  the  issue  of 
providing  feedback  to  the  people  that  proposed/documented  the  issue  and  to  the  people 
who  are  expected  to  implement  the  change.  There  is  an  importance  to  pass  on  feedback 
that  lets  people  know  that  the  issue/problem  is  acknowledged  but  cannot  be  changed  at 
this  time  due  to  various  limitations  such  as  time,  resources,  personnel,  etc. 

Concerning  the  validation  process,  the  Lessons  Learned  Centre  states  that  the  easy 
part  of  a  Lessons  Learned  System  is  identifying  what  the  problem  is. 

While  it  is  one  thing  to  identify  the  deficiency,  it  is  another  step  to  get  people  to 

agree  on  the  solution.86 

The  biggest  challenge  in  a  lessons  learned  organization  is  in  “closing  the  loop”. 
By  “closing  the  loop”  it  is  meant  that  decisions  are  made,  direction  passed  on,  and  change 
is  implemented.  The  Lessons  Learned  Centre  provides  questions  that  if  answered  will 
lead  to  “closing  the  loop”. 

1 .  What  change  might  be  suggested/recommended/considered? 

2.  Who  can  influence  or  initiate  this  change? 

3.  Who  decides  what  change  will  be  initiated  (authority)? 

4.  Who  decides  who  is  responsible  for  the  change? 

5.  Who  decides  when/how  the  change  is  to  be  done  (limits/restrictions)? 

6.  Who  has  authority  to  follow-up  and  ensure  the  change  takes  place? 

The  Army  Lessons  Learned  Centre  has  been  successful. 

We  believe  we  have  been  successful  in  helping  advocate  and  implement  change  in 

the  Army,  however  there  will  always  be  room  for  improvement.  87 

The  consequences  of  disseminating  an  erroneous  or  misleading  lesson  learned 
would  be  poor  information  being  passed  to  the  field,  with  a  resultant  negative  impact  on 
the  Lessons  Learned  organization  and  others  involved  in  passing  on  the  information. 


86  Twohey,  Major.  J.  Mark,  Canadian  Army  Lessons  Learned  Centre  point  of  contact,  (personal 
communication,  e-mail  questionnaire,  January  22,  2002) 

87  Ibid. 
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To  avoid  this  there  is  a  requirement  to  ensure  that  issues  and  lessons  are 
legitimate  and  cover  a  broad  perspective  -  that  is  to  say  they  aren’t  influenced  by  a 
narrow  application  or  a  specific  agenda  by  the  author/initiator.88 

There  is  no  single  person  responsible  for  the  accuracy  of  the  Lessons  Learned 
Centre.  The  Director  of  the  Lessons  Learned  Centre  is  closely  linked  with  the  chain  of 
command  both  in  the  reporting  and  the  change/implementation.  The  Director  has 
responsibilities  to  the  Commander  of  the  Army  regarding  the  Lessons  Learned  Centre. 
As  a  staff  advisor,  the  Director  does  not  have  authority  to  direct  and  implement  change 
unilaterally.  The  Army  Lessons  Learned  Process  involves  more  that  just  the  Army 
Lessons  Learned  Centre. 

2,  The  United  Nations  Peacekeeping  Best  Practice  Unit89 

The  United  Nations  Peacekeeping  Best  Practices  Unit  was  formerly  called  The 
Lessons  Learned  Unit  of  the  Department  of  Peacekeeping  Operations. 

The  Lessons  Learned  Unit  of  the  Department  of  Peacekeeping  Operations  at  the 
United  Nations  was  created  in  1995.  Its  personnel  consisted  of  a  Head  of  the  Unit,  a 
Coordination  Officer,  two  Military  Officers,  two  Research  Analysts,  a  Research  Assistant 
and  an  Administrative  Assistant.90 

The  purpose  or  objectives  of  the  Lessons  Learned  Unit  is  to  draw  lessons  learned 
from  peacekeeping  missions.  It  is  to  recommend  the  application  of  lessons  learned  from 
peacekeeping  missions  to  ongoing  and  future  operations.  It  is  to  monitor  the  application 
of  these  recommendations  and  lessons  learned.  It  is  to  develop  the  Lessons  Learned  Unit 
into  the  United  Nations  institutional  memory  on  peacekeeping  operations  and  to  make 
this  institutional  memory  easily  available  to  officers,  at  Headquarters  and  in  the  field, 

88  Twohey,  Major.  J.  Mark,  Canadian  Army  Lessons  Learned  Centre  point  of  contact,  (personal 
communication,  e-mail  questionnaire,  January  22,  2002) 

89  Reiff,  D,  The  United  Nations  Peacekeeping  Best  Practices  Unit  point  of  contact,  (personal 
communication,  e-mail  questionnaire,  February  1,  2002)  and  Lowe,  S.,  The  United  Nations  Peacekeeping 
Best  Practices  Unit  point  of  contact,  (personal  communication,  telephone.  May  17,  2002) 

90  United  Nations  Lessons  Learned  System  Web  Page 
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involved  in  all  aspects  of  peacekeeping  missions,  including  their  planning,  managing  and 
support  .91 

The  products  of  the  Lessons  Learned  Unit  are  reports.  They  fall  into  two 
categories,  thematic  and  mission  specific.  Thematic  reports  are  more  general.  An 
example  is  Report  of  the  United  Nations  Seminar  on  Public  Information  Policies  and 
Practices  for  Field  Missions  (1997).  Another  example  is  Disarmament,  Demobilization 
and  Reintegration  of  Ex-combatants  in  a  Peacekeeping  Environment;  Principles  and 
Guidelines  (Dec  1999).  An  example  of  a  mission  specific  report  is  Lessons  Learned 
from  the  Angola  Verification  Missions  (UNAVEM  I,  II  and  III):  Interim  Report  (Nov 
1997).  This  report  is  not  published  but  exists  as  an  internal  report. 

The  Lessons  Learned  Unit  considers  there  to  be  two  sources  for  reports.  The 
primary  source  is  first  hand  accounts  by  the  Lessons  Learned  Unit.  Included  in  the 
primary  sources  are  interviews  with  participants  of  a  subject  being  considered  for  lessons 
learned. 

Lessons  were  obtained  from  primary  sources,  such  as  interviews  with  mission  and 
Secretariat  personnel,  representatives  of  specialized  agencies  as  well  as  political 
actors.  Lessons  Learned  teams  visited  mission  areas  to  gather  first  hand 
information  for  mid  and  end  of  mission  assessments. 92 

The  secondary  sources  are  second  hand  accounts  such  as  published  papers. 

The  secondary  sources  of  information  include  published  material,  media  analysis 
and  reportage,  evaluation  reports  of  peacekeeping  operations  by  independent 
experts  and  governments  and  end-of-tour  reports  by  key  personnel,  both  in  the 
field  and  at  Headquarters. 93 

The  Brahimi  report  was  released  in  August  of  2000  after  a  thorough  review  of  UN 
peace  and  security  activities.  Among  the  recommendations  contained  in  that  report  was 
that  the  Lessons  Learned  Unit  should  be  located  where  it  could  work  more  closely  with 
and  contribute  effectively  with  ongoing  operations,  as  well  as  mission  planning  and 
doctrine/ guide  lines  development.  At  that  time  the  Lessons  Learned  Unit  merged  with  the 

91  United  Nations  Lessons  Learned  System  Web  Page 

92  Ibid. 

93  Ibid. 
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Policy  and  Analysis  Unit  to  form  what  is  now  called  the  Peaeekeeping  Best  Practiees 
Unit. 

As  part  of  implementing  the  Brahimi  report,  the  methodology  used  for  extracting 
and  validating  lessons  learned/best  praetiees  is  currently  under  review.  It  is  expected 
that  the  extracting  and  validating  of  lessons  learned  in  the  future  will  be  similar  to  the 
past  system.  The  past  method  of  extraeting  lessons  was  described  above,  that  being 
primary  and  secondary  sources. 

The  method  of  validation  was  as  follows.  A  first  draft  of  a  report  was  written  by 
the  Lessons  Learned  Unit.  The  authors  then  resided  over  an  internal  UN  review  of  the 
draft  called  an  expert  workshop.  The  membership  of  the  expert  workshop  consisted  of 
departments  of  the  UN  and  different  levels  of  position.  For  example  the  Peaee  Keeping 
department  and  the  Humanitarian  department,  among  others,  are  represented  and  the 
representation  eonsists  of  policy-making  positions  as  well  as  lower  management.  The 
expert  workshop  reaehes  common  agreement  with  the  lessons  learned  and  possible  policy 
change  or  agrees  to  disagree.  The  authors  of  the  report  have  final  say  on  the  contents  of 
the  first  draft. 

However  one  purpose  of  the  Lessons  Learned  Unit  has  been  satisfied.  That  is 
providing  information  on  lessons  learned  to  policy  makers  who  can  implement  the 
lessons  learned.  This  is  done  by  the  poliey  makers  participating  in  the  validation  proeess 
through  the  expert  workshop. 

From  the  expert  workshop  the  draft  report  goes  through  an  external  review.  This 
ineludes  member  states.  Member  states  comment  on  the  report  but  again  the  authors 
have  the  final  say.  There  is  no  requirement  for  the  member  states  to  sign  up  to  the  report 
and  as  sueh  the  report  is  not  an  official  UN  paper. 
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3. 


American  Industrial  Hygiene  Association  (AIHA)  Laboratory  Health 
&  Safety  Committee94 


The  American  Industrial  Hygiene  Association  Laboratory  Health  &  Safety 
Committee  is  located  in  Fairfax,  Virginia.  Its  mission  statement  is: 

To  provide  a  forum  on  the  practice  of  industrial  hygiene  and  safety  in  the 
laboratory  and  associated  research  and  support  service  settings  and  to  participate 
in  the  development  and  analysis  of  related  technological  and  regulatory  issues. 95 

To  support  this  mission  statement,  a  lessons  learned  system  has  been  established. 
The  goal  of  the  lessons  learned  portion  of  the  Health  &  Safety  Committee  is  to  collect 
lessons  learned  as  a  result  of  laboratory  mishaps.  A  primary  source  of  these  lessons 
learned  are  University  laboratories.  Also  included  in  the  lessons  learned  system  goal  is 
making  the  mishaps  available  to  others  so  that  mishaps  are  not  repeated. 

Lessons  are  obtained  by  advertisement  for  input  on  their  web  page.  There  is  an 
electronic  form  that  is  made  available  for  anyone  to  submit  information  about  an  incident. 
It  is  requested  that  the  submission  include  not  only  an  account  of  the  mishap  but  realized 
“key  safety  concepts  and  principles”  and  include  a  corrective  action.  Other  sources  of 
lessons  obtained  from  laboratory  mishaps  are  those  presented  informally  through  contact 
with  the  committee  members.  Also  there  is  a  regular  communication  that  exists  with 
college  laboratories  that  provide  lessons  from  mishaps  as  well  as  communication  with 
industrial  laboratories. 

The  way  lessons  are  considered  acceptable  for  publishing  is  the  submission  is 
reviewed  by  a  group  of  two  or  three  committee  members  and  if  there  appears  to  be  a 
lesson  to  be  learned  from  the  mishap  then  the  submission  is  published  on  their  web  site. 
Identifying  information  on  people  and  facilities  is  not  included  in  the  publication.  Also, 
no  corrective  action  or  interpretation  is  offered,  just  the  story.  Publication  includes  “key 
safety  concepts  and  principles”  and  suggested  corrective  action. 


94  Krethman,  K.,  American  Industrial  Hygiene  Association  Laboratory  Health  &  Safety 
Committee  point  of  contact  (personal  communication,  e-mail  questionnaire,  January  21,  2002) 

95  American  Industrial  Hygiene  Association  Laboratory  Health  &  Safety  Committee  Web  Page 
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The  committee  considers  the  lessons  learned  system  to  be  successful.  The 
mishaps  are  published.  This  meets  the  basic  goal  of  collecting  and  publishing  laboratory 
mishaps.  It  is  a  good  collection  of  mishaps  and  they  are  well  presented. 

There  is  not  one  person  responsible  for  the  accuracy  of  the  lessons  learned 
system.  There  is  a  disclaimer  associated  with  the  published  incidents.  The  disclaimer 
states  that  the  safety  committee  does  not  take  any  responsibility  for  the  accuracy  of  the 
incidents  nor  does  any  view  necessarily  reflect  the  views  of  the  committee.96 

The  AIHA  Laboratory  Health  &  Safety  Committee  is  in  the  developing  stage.  Its 
workforce  consists  of  volunteer  committee  members  and  its  financial  needs  for  operation 
are  not  great.  Participation  after  initial  startup,  in  terms  of  submission  of  lessons,  has 
leveled  off  or  been  slow.  Most  submissions  of  lessons  have  been  from  those  sought 

out.97 


4,  U.S.  Army  Center  for  Engineer  Lessons  Learned98 

The  purpose  of  the  U.S.  Army  Center  for  Engineer  Lessons  Learned  is  to  collect, 
analyze  and  incorporate  Engineer  lessons  learned.  Some  examples  of  the  subject  of  these 
lessons  learned  are  the  performance  of  vehicles  and  the  operation  of  mine  clearing 
devices.99  The  U.S.  Army  Center  for  Engineer  Lessons  Learned  is  separate  from  the 
Center  for  Army  Lessons  Learned  but  collaborates  with  the  latter  whenever  possible. lOO 

Lessons  are  obtained  in  a  few  ways.  In  a  large-scale  operation,  the  Center  for 
Army  Lessons  Learned  will  send  out  a  Combined  Arms  Assessment  Team  to  observe  and 

96  American  Industrial  Hygiene  Association  Laboratory  Health  &  Safety  Committee  Web  Page 

97  Krethman,  K.,  American  Industrial  Hygiene  Association  Laboratory  Health  &  Safety 
Committee  point  of  contact  (personal  communication,  e-mail,  July  9,  2002) 

98  Snodgrass,  R.,  U.  S.  Army  Center  for  Engineer  Lessons  Learned  point  of  contact,  (personal 
communication,  e-mail  questionnaire,  January  24,  2002) 

99  Snodgrass,  R.,  U.  S.  Army  Center  for  Engineer  Lessons  Learned  point  of  contact,  (personal 
communication,  telephone,  July  16,  2002) 

100  Snodgrass,  R.,  U.  S.  Army  Center  for  Engineer  Lessons  Learned  point  of  contact,  (personal 
communication,  telephone,  July  10,  2002) 
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interview  so  as  to  collect  lessons  learned.  There  are  instances  when  a  member  of  the 
team  is  part  of  Army  Center  for  Engineer  Lessons  Learned.  Appropriate  lessons  are  then 
brought  to  the  Army  Center  for  Engineer  Lessons  Leaned.  Another  way  is  similar  except 
the  Combined  Arms  Assessment  Team  performs  observation  and  interviews  at  the 
Combined  Arms  Training  Center. 

In  smaller  operations,  on-going  operations,  and  other  training  missions,  the  units 
involved  will  provide  lessons  learned  directly  to  the  Army  Center  for  Engineer  Lessons 
Learned.  The  vehicle  often  used  is  an  After  Action  Report  from  the  participating  units. 

The  method  of  validating  the  lessons  learned  is  done  at  the  time  of  collection. 
The  Combined  Arms  Assessment  Team  confirms  the  observation  with  the  unit  for 
accuracy.  When  the  unit  submits  an  After  Action  Report,  the  After  Action  Report  is 
considered  accurate,  as  the  After  Action  Report  is  what  is  used  to  brief  their  higher 
headquarters.  The  After  Action  Report  is  also  validated  by  comparing  it  with  previous 
information  in  the  specific  area.  If  there  is  some  discrepancy  the  After  Action  Report 
will  be  rechecked  with  the  submitting  unit. 

The  validation  continues  with  subject  matter  experts  reviewing  the  lessons 
learned.  If  it  is  necessary  they  will  perform  tests  to  ensure  the  information  is  correct. 
Lrom  there  the  lessons  learned  are  distributed  to  the  appropriate  place. 

The  Army  Center  for  Engineer  Lessons  Learned  considers  itself  successful. 
Lessons  learned  input  has  effected  changes  to  doctrine,  training  and  equipment.  There 
are  serious  consequences  for  disseminating  erroneous  or  misleading  lessons  learned.  The 
consequences  could  be  mission  failure,  injury  or  equipment  failure. 

The  Army  Center  for  Engineer  Lessons  Learned  is  in  the  development  stage  and 
receiving  lessons  learned  is  its  present  concern.  Although  the  method  of  lessons 
gathering  involves  some  active  sourcing,  the  system  is  mostly  passive  in  this  regard  and 
there  is  a  concern  that  there  are  lessons  learned  that  exist  in  the  field  that  are  not  reaching 
The  Army  Center  for  Engineer  Lessons  Learned  system,  loi 


101  Snodgrass,  R.,  U.  S.  Army  Center  for  Engineer  Lessons  Learned  point  of  contact,  (personal 
communication,  telephone,  July  10,  2002) 
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5,  Army  Medical  Department  Lessons  Learnedio^ 

The  purpose  of  the  Army  Medical  Department  Lessons  Learned  is  to  collect, 
analyze  and  disseminate  US  medical  unit  experiences  and  lessons  learned.  It  existed  in 
some  form  in  1991  but  its  present  form  has  its  beginnings  in  1998  so  it  is  relatively  new. 
The  Army  Medical  Department  Lessons  Learned  is  separate  from  the  Center  for  Army 
Lessons  Learned  although  there  is  collaboration  whenever  possible.  103 

Lessons  are  obtained  by  unit  or  individual  submissions.  By  definition,  the 
submissions  contain  observations  or  issues;  they  are  not  considered  lessons  learned  at  this 
point  in  the  process.  The  unit  observations  or  issues  are  analyzed  by  the  Army  Medical 
Department  Lessons  Learned  office  and  forwarded  to  the  appropriate  subject  matter 
proponent  for  validation  and  verification.  Based  on  the  subject  matter  expert  analysis  and 
proponent  verification,  the  Army  Medical  Department  Center  staff  directs  work  on  a 
solution  by  the  Army  Medical  Department  Center  and  School.  The  proponent  then 
validates  the  solution. 

This  is  a  formal  process  and  the  reason  this  process  is  formal  is  to  ensure  that 
recommendations  are  appropriate  for  the  observation.  Everything  that  is  submitted  is 
reviewed  but  not  everything  is  forwarded  to  the  proponent  office  for  action. 

In  some  cases,  a  Combat  Training  Center  focused  rotation  is  used  to  test  and 
validate  a  new  concept  or  a  solution  to  one  of  the  observations  developed  by  the  Army 
Medical  Department  Center  and  School.  In  other  cases,  a  unit  may  volunteer  to  test  the 
new  concept  or  solution.  There  is  an  Army  Medical  Department  Lessons  Learned  Board 
that  monitors  if  the  solution  works.  If  the  solution  does  not  work,  then  the  proponent 
staff  begins  again.  If  a  solution  is  obtained,  then  the  solution  is  validated  and  the  solution 
becomes  a  Lesson  Learned.  A  Lesson  Learned  is  defined  as  an  Army  Medical 
Department-wide  change  as  a  result  of  a  submitted  observation. 

102  Rathbun,  G.,  Army  Medical  Department  Lessons  Learned  point  of  contact,  (personal 
communication,  e-mail  questionnaire,  January  27,  2002) 

103  Rathbun,  G.,  Army  Medical  Department  Lessons  Learned  point  of  contact,  (personal 
communication,  telephone,  July  10,  2002) 
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The  Army  Medical  Department  Lessons  Learned  has  been  successful.  Here  is  an 
example.  A  Combat  Training  Center  observation  was  Medical  unit  leaders  are  not 
proficient  in  battle  tracking.  104  This  was  validated  as  an  observation  over  three  Combat 
Training  Centers  and  several  rotations.  The  solution  was  to  add  a  battle  tracking  course 
to  the  Officer  Basic  Course.  The  National  Tracking  Center  has  verified  that  battle 
tracking  has  improved. 

Lessons  Learned  effect  the  entire  Army  Medical  Department.  They  become 
Army  Medical  Department  Doctrine,  Mission  Training  Plans  and  Programs  of  Instruction 
and  other  products  for  which  the  Army  Medical  Department  Lessons  Learned  Center  and 
School  are  proponents.  Because  of  this,  there  are  few  Lessons  Learned  compared  to 
observations. 

The  present  concern  is  obtaining  lessons  learned.  Although  active  observations 
occur  there  is  a  reliance  on  input  from  field  activities.  These  field  activities  do  not  hold 
supporting  the  Army  Medical  Department  Lessons  Learned  Center  as  a  high  priority  and 
there  is  no  penalty  for  the  omission.  There  is  also  missed  opportunity.  For  example, 
there  is  no  Center  for  Army  Lessons  Learned  representatives  in  Afghanistan.! 05 

It  was  noted  that  the  Navy  is  starting  a  program  similar  to  the  Army  Medical 
Department  Lessons  Learned  Center  with  preliminary  acronym  of  NOMI.  This  will 
allow  collaboration,  which  is  most  appropriate. 


6,  Coast  Guard  -  Standard  After  Action  Information  and  Lessons 
Learned  Systemi06 

The  purpose  of  the  Coast  Guard  -  Standard  After  Action  Information  and  Lessons 
Learned  System  is  to  capture  After  Action  Reports,  Lessons  Learned  and  Best  Practices. 
It  is  to  share  this  information  amongst  Coast  Guard  Commands  and  to  other  Federal 


104  Combat  Training  Center  observation 

105  Rathbun,  G.,  Army  Medical  Department  Lessons  Learned  point  of  contact,  (personal 
communication,  telephone,  July  10,  2002) 

106  Burt,  M.,  Coast  Guard  -  Standard  After  Action  Information  and  Lessons  Learned  System 
point  of  contact,  (personal  communication,  e-mail  questionnaire,  January  24,  2002) 
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Agencies  as  necessary.  It  is  to  have  this  information  to  enhance  unit  preparedness, 
readiness  and  training.  It  was  established  in  October  2001 . 

The  Coast  Guard  -  Standard  After  Action  Information  and  Lessons  Learned 
System  is  accessible  on  the  Coast  Guard  Intranet.  Lessons  for  the  Coast  Guard  - 
Standard  After  Action  Information  and  Lessons  Learned  System  are  obtained  by  Coast 
Guard  Units  linking  to  the  site  and  entering  a  lesson  learned.  Lessons  Learned  are  also 
released  to  the  Coast  Guard  -  Standard  After  Action  Information  and  Lessons  Learned 
System  by  Coast  Guard  Headquarters.  Coast  Guard  Units  link  to  the  site  to  review 
posted  lessons  learned. 

To  provide  validation  to  the  lessons  learned  submitted  to  the  Coast  Guard  - 
Standard  After  Action  Information  and  Lessons  Learned  System,  structural  guidance  is 
provided  to  the  units  that  reports  entered  into  the  Coast  Guard  Standard  After  Action 
Information  and  Lessons  Learned  System  are  considered  Command  approved.  There  is 
also  a  non-mandatory  request  to  include  the  name  of  the  person  or  unit  entering  the 
lesson  learned.  This  provides  some  check  that  lessons  learned  submitted  are  authorized 
and  Command  approved.  The  Coast  Guard  -  Standard  After  Action  Information  and 
Lessons  Learned  System  Administrator  reviews  the  submitted  lessons  learned  before 
posting  on  the  Coast  Guard  -  Standard  After  Action  Information  and  Lessons  Learned 
System  Intranet  website.  Some  lessons  learned  submitted  pertaining  to  certain  subjects 
are  required  to  be  forwarded  to  Coast  Guard  Headquarters  for  review. 

The  Coast  Guard  Headquarters  reviews  these  lessons  learned.  It  also  reviews 
reports  generated  by  Coast  Guard  Units  that  have  been  reviewed  and  approved  through 
the  Chain  of  Command  leading  to  Coast  Guard  Headquarters.  Abstracted  from  review  of 
the  lessons  learned  and  reports  is  information  that  is  used  to  prepare  releases  to  the  Coast 
Guard  -  Standard  After  Action  Information  and  Lessons  Learned  System  to  be  published 
on  the  Intranet  website. 

The  main  users  of  the  Coast  Guard  -  Standard  After  Action  Information  and 
Lessons  Learned  System  are  Coast  Guard  Unit  contingency  planners.  An  erroneous  or 
misleading  report  could  effect  the  efficiency  of  the  Coast  Guard  to  perform  its  mission. 
The  Command  Approval  and  reviews  minimizes  the  possibility  that  erroneous  or 
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misleading  reports  are  placed  into  the  Coast  Guard  -  Standard  After  Action  Information 
and  Lessons  Learned  System  Intranet  website.  There  is  also  the  ability  to  pull  back  off 
the  website  a  lesson  learned  that  is  erroneous  or  misleading. 

The  Coast  Guard  -  Standard  After  Action  Information  and  Lessons  Learned 
System  has  been  successful.  This  is  based  on  the  number  of  lessons  learned  received. 
This  is  also  based  on  the  usage  of  the  Coast  Guard  Standard  After  Action  Information 
and  Lessons  Learned  System  Intranet  website.  This  initial  success  is  guarded  however 
and  the  present  concern  is  for  there  to  be  a  continued  contribution  of  lessons  learned  from 
the  field,  as  this  is  not  a  requirement. 


7.  Best  Manufacturing  Practices  Programios 

The  Best  Manufacturing  Practices  Program  is  sponsored  by  the  Office  of  Naval 
Research.  It  was  created  in  1985  to  overcome  the  wide  and  very  costly  variances  in  the 
quality  of  goods  and  services  being  received  by  the  Navy  from  contractors  throughout  the 
United  States.  109  Navy  contractors  voluntarily  agree  to  share  their  solutions  to 
manufacturing  process  problems  still  being  experienced  by  other  Navy  contractors.  The 
Best  Manufacturing  Practices  Program  provides  the  data  gathering,  validation  and 
dissemination.  The  goal  of  the  Best  Manufacturing  Practices  Program  is  to  improve  the 
quality,  cost,  and  reliability  of  goods  and  services  the  Navy  receives. 

Manufacturing  processes  as  defined  above  includes  technical  and  administrative. 
For  example  a  best  practice  may  be  to  use  integrated  teams  consisting  of  multiple 
disciplines  in  designing  a  product.  A  company  may  decide  to  implement  this  best 
practice.  In  implementing  this  best  practice,  the  lessons  learned  are  recorded.  Included  in 
the  writings  of  the  Best  Practices  database  of  the  website  are  listed  the  lessons  learned,  no 

10^  Burt,  M.,  Coast  Guard  -  Standard  After  Action  Information  and  Lessons  Learned  System 
point  of  contact,  (personal  communication,  telephone,  July  10,  2002) 

108  Robertson,  L.,  Best  Manufacturing  Practices  point  of  contact,  (personal  communication,  e- 
mail  questionnaire,  February  7,  2002) 

109  Halbig,  L.,  Description  of  Best  Manufacturing  Practices  Program,  provided  January  24,  2002 

1 10  Best  Manufacturing  Web  Page 
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The  acceptance  of  best  practices  is  done  very  formally  by  the  use  of  survey  teams. 
A  participating  company  notifies  the  Best  Manufacturing  Practices  Program  of  a  best 
practice  that  the  company  is  willing  to  share.  A  pre-survey  team  visits  the  company  to 
obtain  preliminary  information  and  plan  for  a  future  visit  by  a  formal  survey  team 
appropriate  to  the  subject  matter  if  they  consider  the  best  practice  worth  pursuing.  The 
formal  survey  team  comprised  of  impartial  experts  from  government,  industry,  and 
academia  visits  the  company  and  documents  what  they  feel  qualifies  for  a  best  practice. 
The  best  practice  is  then  disseminated  through  the  Best  Manufacturing  Practices  Program 
Internet  website.  It  is  reviewed  for  technical  accuracy  by  the  surveyed  company  before  it 
is  released.  Often  times  the  release  includes  information  about  a  new  product  or  process 
of  the  company. 

The  purpose  of  the  Lessons  Learned  portion,  included  in  the  writings  located  in 
the  Best  Practices  database  in  the  Best  Manufacturing  Practices  Program  Internet  website 
is  to  make  others  aware  of  some  of  the  pitfalls  that  the  company  implementing  the  Best 
Practice  encountered  so  that  another  company  or  activity  implementing  the  same  or 
similar  practice  does  not  repeat  the  same  mistake.m 

Lessons  are  obtained  by  inclusion  within  write-ups  by  companies  who  are 
participating  in  a  Best  Manufacturing  Practices  Program  review  of  the  Best  Practice.  The 
validation  of  the  Best  Practice,  as  described  above,  is  very  formal  but  there  is  no 
validation  of  the  lessons  learned. 

No  measure  of  success  for  the  Lessons  Learned  portion  has  been  attempted.  The 
Best  Manufacturing  Practices  Program  has  been  successful,  being  the  winner  of  the 
Innovations  in  American  Government  Award  and  the  Vice  President’s  Hammer  Award. 


1 1 1  Robertson,  L.,  Best  Manufacturing  Practices  point  of  contact,  (personal  communication,  e- 
mail  questionnaire,  February  7,  2002) 
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8,  Department  of  Energy  Project  Hanford  Lessons  Learned  Systemii2 

The  Project  Hanford  Lessons  Learned  System  serves  the  Hanford  Site  located 
along  the  Columbia  River  in  southeastern  Washington  State.  Hanford  produced 
plutonium  for  the  Manhattan  Project  during  World  War  II  and  the  Cold  War  and 
is  now  undergoing  environmental  restoration  under  the  Department  of  Energy.  113 

The  Department  of  Energy  Project  Hanford  Eessons  Eeamed  System  assumed  its 
present  form  in  1994  and  is  a  mature  system. 

The  purpose  of  the  Project  Hanford  Eessons  Eearned  System  is  to  publicize  good 
work  practices  so  others  can  adopt  them  to  improve  efficiency  and  performance  and  to 
share  lessons  learned  arising  from  accidents  so  that  others  can  avoid  making  the  same  or 
similar  errors.  114 

A  lesson  learned  for  the  Project  Hanford  Eessons  Learned  System  is  defined 
consistent  with  DOE  Standard  7501-99  December  99.  Eessons  are  obtained  for  the 
Project  Hanford  Eessons  Eearned  System  in  the  following  way.  Each  day,  the  Eessons 
Eeamed  Coordinator  screens  the  Department  of  Energy  Occurrence  Reporting  and 
Processing  System  for  events  across  the  Department  of  Energy  Complex  that  could  also 
happen  at  the  Hanford  site.  These  events  become  input  into  a  process  that  could  lead  to  a 
lessons  learned  at  the  Hanford  site.  Also  included  as  input  when  deemed  appropriate  are 
Hanford  site  items  from  the  corrective  action  management  group. 

A  list  server  also  provides  lessons  learned  from  other  Department  of  Energy 
Eessons  Eeamed  Systems.  These  lessons  learned  are  then  passed  to  the  appropriate 
Hanford  Site  management  for  action  as  appropriate. 


1 12  Bickford,  J.,  Department  of  Energy  Project  Hanford  Lessons  Learned  System  point  on 
contact,  (personal  communication,  e-mail  questionnaire,  January  24,  2002) 

113  Bickford,  J.,  Department  of  Energy  Project  Hanford  Lessons  Learned  System  point  on 
contact,  (personal  communication,  e-mail,  questionnaire,  July  9,  2002),  see  also 
http://www.hanford.gov/rl/siteinfo/knowus.asp 

1 14  Bickford,  J.,  Department  of  Energy  Project  Hanford  Lessons  Learned  System  point  on 
contact,  (personal  communication,  e-mail  questionnaire,  January  24,  2002) 
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A  draft  lesson  learned  is  prepared  from  the  inputs  mentioned  previously  by  the 
Lessons  Learned  Coordinator.  The  draft  lesson  learned  is  then  e-mailed  to  subject  matter 
experts,  Lessons  Learned  Point-of-Contact  at  the  originating  facility  if  applicable,  the 
originator  if  the  event  is  from  the  Hanford  Site  and  management  as  appropriate.  A  set 
time  is  given  to  provide  comments  otherwise  it  is  considered  concurred  with.  The 
comments  are  incorporated  and  possibly  sent  out  as  a  draft  again.  When  there  is 
concurrence,  the  draft  becomes  a  lesson  learned  and  is  entered  into  the  Project  Hanford 
Lessons  Learned  System. 

The  Project  Hanford  Lessons  Learned  System  has  been  a  success. 

Several  prevented  or  mitigated  accidents  can  be  traced  directly  to  the  Project 

Hanford  Lessons  Learned  System. ii5 

The  overall  accident/injury  rate  has  also  decreased  over  the  last  seven  years  and 
this  is  partly  due  to  the  Project  Hanford  Lessons  Learned  System.  Also  many  of  the 
Project  Hanford  Lessons  Learned  System  good  work  practices  have  been  implemented 
leading  to  more  efficient  operation  at  the  Hanford  site. 

It  was  noted  that,  in  general,  the  consequences  of  disseminating  erroneous  or 
misleading  lessons  learned,  aside  from  the  consequences  related  to  implementation  of  the 
misleading  lesson  learned,  is  an  erosion  in  the  credibility  of  any  Lessons  Learned  System 
and  the  associated  reduction  in  its  value.  For  this  reason,  the  Lessons  Learned 
Coordinator,  with  reliance  on  subject  matter  experts,  assumes  full  responsibility  for  the 
accuracy  of  the  Project  Hanford  Lessons  Learned  System,  which  is  an  exemplary 
example  of  a  quality  Lessons  Learned  System. 

A  few  years  ago.  General  Motors  invited  the  Hanford  Lessons  Learned 
Coordinator  to  provide  guidance  for  their  quality  improvement  initiative.  One  suggestion 
was  to  keep  the  Lessons  Learned  System  simple  and  familiar. 


1 15  Bickford,  J.,  Department  of  Energy  Project  Hanford  Lessons  Learned  System  point  on 
contact,  (personal  communication,  e-mail  questionnaire,  January  24,  2002) 
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Make  the  system  fit  within  the  tools  your  workers  use  every  day  so  they  do  not 
need  to  learn  something  new.  If  they  use  Lotus  Notes,  build  your  system  in  that 
suite.  If  your  business  uses  an  intranet  extensively,  use  that.  If  e-mail  is  the 
eommunieation  medium  of  ehoiee,  send  lessons  learned  by  e-mail. 116 

Other  suggestions  included  the  need  for  management  support  at  all  levels  and 
tailoring  the  distribution  to  the  user.  The  Hanford  Lessons  Learned  System  is  a  mature 
Lessons  Learned  System  and  its  success  can  probably  be  attributed  to  its  methods  and 
personnel. 


9.  BNFL  Incorporated  Lessons  Learned  Systemii7 

BNFL  Incorporated  is  a  wholly  owned  subsidiary  of  British  Nuclear  Fuels 
Limited.  BNFL  Incorporated  provides  decontamination  and  decommissioning  resources. 
BNFL  Incorporated  holds  a  contract  with  the  Department  of  Energy  to  remove  equipment 
and  decontaminate  three  huge  former  process  buildings  at  the  Department  of  Energy  Oak 
Ridge  Tennessee  Technology  Park.ns 

The  purpose  of  the  BNEE  Incorporated  Eessons  Teamed  System  is  to  identify 
good  practices  within  BNEE  and  the  Department  of  Energy  and  to  provide  these  practices 
to  the  current  project  at  Oak  Ridge  for  implementation.  It  is  also  to  identify  poor  work 
practices  within  BNEE  so  they  will  not  be  repeated  and  poor  work  practices  within  the 
Department  of  Energy  so  they  can  be  avoided. 

Eessons  are  obtained  through  the  Department  of  Energy  Eist  Server,  through 
internal  BNEE  events  and  through  on  site  events.  On  site  events  and  BNEE  corporate 
events  are  analyzed  for  causes  and  lessons  are  developed  based  on  a  causal  analysis. 

To  the  BNEE  Incorporated  Eessons  Teamed  System,  verification  is  the  act  of 
ensuring  that  a  lesson  was  developed  for  an  event  and  distributed.  Validation  is  the  act  of 

1 16  Bickford,  J.,  Department  of  Energy  Project  Hanford  Lessons  Learned  System  point  on 
contact,  (personal  communication,  e-mail,  questionnaire,  July  9,  2002} 

1 17  Cooler,  M.,  BNFL  Inc.  Lessons  Learned  System  point  of  contact,  (personal  communication, 
e-mail  questionnaire,  January  29,  2002) 

118  BNFL  Web  Page 
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ensuring  that  the  lesson  effectively  addressed  the  event  and  the  corrective  action 
prevented  recurrence  of  the  event.  The  degree  of  formality  used  in  validation  and 
verification  depends  on  the  significance  of  the  event.  If  the  significance  has  potentially 
serious  consequences  such  as  loss  of  life,  injury  to  multiple  workers  or  adverse 
environmental  consequences,  then  the  degree  of  formality  is  high.  When  the  event  is 
positive,  such  that  it  would  generate  a  good  work  practice,  the  events  are  rarely  validated. 

The  validation  process  consists  of  a  reviewer  or  a  set  of  reviewers  monitoring  for 
the  precursors  of  the  event  that  initiated  the  lesson  learned.  This  is  done  over  a  period 
of  time.  If  the  event  or  its  precursors!  19  do  not  occur  then  it  is  concluded  that  the  lesson 
learned  included  an  accurate  corrective  action.  If  the  event  is  repeated,  for  example  if 
there  is  a  repeat  of  inadvertent  disconnection  of  electrical  lines,  or  repeats  of  similar 
nature  such  as  repeat  incidents  of  insufficiently  trained  personnel  making  work  control 
errors,  then  there  is  a  new  analysis,  a  new  lesson  and  a  new  set  of  actions. 

The  success  of  the  BNFL  Incorporated  Lessons  Learned  System  has  been  mixed. 
There  are  a  vast  number  of  events  that  are  transformed  into  a  lesson  learned.  Issuing  the 
lessons  through  the  BNFL  Incorporated  Lessons  Learned  System  is  easy;  ensuring  their 
appropriate  incorporation  in  work  plans  is  more  difficult.  If  the  initiating  event  is  serious, 
then  incorporation  is  most  likely.  The  easiest  lessons  to  enforce  are  those  relating  to 
product  failures  or  recalls.  Lessons  based  on  events  that  led  to  curtailing  of  activities  are 
also  usually  implemented.  Positive  practices  leading  to  increased  efficiency  are  given  the 
least  attention  by  implementers. 

The  consequences  of  disseminating  an  erroneous  or  misleading  lesson  learned  is 
dependent  of  the  seriousness  of  the  originating  event.  It  was  also  pointed  out  that  there 
would  be  a  lessened  reliance  on  the  BNFL  Incorporated  Lessons  Learned  System  if  it 
were  sometimes  inaccurate  with  lessons.  In  order  to  expedite  lessons  but  not  reduce 
quality,  which  may  be  time  consuming,  some  lessons  are  released  as  an  alert  with  the 
statement  “this  alert  is  based  on  immediately  available  information  and  will  be  updated  as 


119  Cooter,  M.,  BNFL  Inc.  Lessons  Learned  System  point  of  contact,  (personal  communication, 
e-mail,  July  17,  2002) 
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further  investigation  is  completed.”i20  The  BNFL  Ineorporated  Lessons  Learned  System 
puts  a  very  high  priority  on  aecuracy  versus  quantity. 


10,  Department  of  Energy  Headquarters  Lessons  Learned  Systemi^i 

The  United  States  Department  of  Energy  has  over  one  hundred  different 
laboratories  and  contractors  involved  in  thousands  of  activities.  The  Department  of 
Energy  Headquarters  Eessons  Eearned  System  provides  a  central  location  for  efficient 
searches  of  valuable  Eessons  Eearned  information.  This  information  can  be  divided  into 
two  categories.  One  category  is  information  on  events  that  occurred  at  Department  of 
Energy  sites  and  the  analysis  of  which  can  lead  to  operational  benefits  at  the  site.  The 
second  category  is  to  provide  guidance  and  information  on  Eessons  Eearned  Systems 
themselves. 

The  Purpose  of  the  Department  of  Energy  Headquarters  Eessons  Eearned  System 
is  to  facilitate  continuous  and  systematic  information  sharing  and  learning  across  the 
Department  of  Energy  Complex.  This  is  to  promote  safety,  cost  effectiveness,  greater 
efficiency,  better  operational  results  and  fewer  mistakes.  Costs  are  reduced  by  providing 
information  on  success  stories  that  if  implemented  would  lead  to  increased  efficiency  at  a 
Department  of  Energy  site.  Costs  are  also  reduced  by  providing  information  on  costly 
mistakes  that  could  be  avoided.  The  purpose  is  also  to  connect  other  sites  with  experts 
doing  similar  work  for  their  experiences.  The  Department  of  Energy  Headquarters 
Eessons  Eearned  System  also  provides  Eessons  Eearned  resources  such  as  information  on 
publications,  conferences  and  workshops  relating  to  Eessons  Eearned  Systems. 

Eessons  are  obtained  in  multiple  ways.  One  way  is  by  conducting  critiques  after 
an  accident.  Another  is  through  procedures  for  performing  work  activities.  A 
requirement  of  the  procedure  is  documenting  what  went  well,  what  did  not  go  well  and 
feeding  the  information  back  to  the  work  planner  to  adjust  the  work  packages.  This  is 

DO  Cooter,  M.,  BNFL  Inc.  Lessons  Learned  System  point  of  contact,  (personal  communication, 
e-mail  questionnaire,  January  29,  2002) 

121  Breslau,  B.  A.,  Department  of  Energy  Headquarters  Lessons  Learned  System  point  of  contact, 
(personal  communication,  e-mail  questionnaire,  January  24,  2002) 
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more  applicable  to  local  activities  but  Lessons  Learned  are  entered  into  the  Department 
of  Energy  Headquarters  Lessons  Learned  System.  Also  as  part  of  any  activity,  on 
completion  of  a  project,  for  a  process  and  program  review,  lessons  are  obtained. 

Each  Department  of  Energy  component  has  its  Eessons  Eeamed  System  that  is 
run  by  a  Eessons  Eeamed  Coordinator.  The  Coordinator  facilitates  the  capture  and 
dissemination  of  information.  The  Coordinator  relies  on  subject  matter  experts  to  assist 
in  preparing  a  lesson  learned  such  that  it  will  be  technically  accurate.  Anyone  may 
submit  a  lessons  learned  or  good  work  practice.  The  submittal  goes  through  the  Eessons 
Eeamed  Coordinator  who  will  pass  it  to  various  departments  such  as  maintenance, 
research  and  development,  or  training  as  appropriate.  In  some  cases  the  lesson  is 
reviewed  by  the  subject  matter  experts  to  insure  technical  accuracy  before  dissemination. 

The  Department  of  Energy  Headquarters  Eessons  Eeamed  System  has  been 
successful.  There  are  many  examples  where  information  provided  to  an  organization 
improved  efficiency  or  prevented  a  recurrence  of  an  accident.  The  system  is  not  perfect 
however.  There  are  cases  where  lessons  learned  information  was  received  by  an 
organization  but  not  acted  on  resulting  in  the  recipient  suffering  the  same  consequences 
as  the  group  providing  the  lesson  learned. 


11.  Department  of  Energy  Office  of  Environmental  Management  Lessons 
Learned  Programt22 

The  Department  of  Energy  Office  of  Environmental  Management  Eessons 
Eeamed  Program  promotes  the  sharing  of  knowledge  across  the  Department  of  Energy  - 
Environmental  Management  complex  with  an  emphasis  on  lessons  learned  relevant  to 
environmental  management  business  and  functional  areas.  It  was  established  in  1996  and 
is  a  somewhat  mature  although  still  developing  Lessons  Learned  System. 

The  tools  used  in  the  collection  and  dissemination  of  lessons  learned  include  the 
Department  of  Energy  Office  of  Environmental  Management  Eessons  Eeamed  Program 
website,  the  on-line  Eessons  Eeamed  database  and  the  Department  of  Energy  Eistserver. 

122  McCune,  M.,  Department  of  Energy  Office  of  Environmental  Management  Lessons  Learned 
Program  point  of  contact,  (personal  communication,  e-mail  questionnaire,  February  2,  2002) 
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Lessons  are  obtained  in  a  number  of  ways.  One  way  is  through  the  Department  of 
Energy’s  formal  oecurrence  reporting  system.  Another  is  through  the  submittal  of 
lessons  learned  on  the  Department  of  Energy  Office  of  Environmental  Management 
Eessons  Eearned  Program  web  page.  This  is  how  most  Department  of  Energy  sites  send 
their  lessons.  Eessons  are  also  obtained  through  subscription  to  other  offices  or  agencies’ 
listservers  and  actively  seeking  lessons  at  meetings  and  workshops  such  as  the  Technical 
Information  Exchange  Workshop. 

The  degree  of  formality  in  the  Department  of  Energy  Office  of  Environmental 
Management  Eessons  Eearned  Program  depends  on  the  field  office  from  where  it 
originates.  Each  field  office  has  its  own  management  system  for  validation  of  the  lesson. 
They  are  also  reviewed  at  the  field  level  public  relations  department  to  insure  the  lesson 
learned  does  not  contain  any  classified  information.  When  the  lessons  learned  are 
received  at  the  Department  of  Energy  Office  of  Environmental  Management  Eessons 
Eearned  Program  they  are  given  a  cursory  check  to  make  sure  the  lesson  reads  well  and 
that  all  the  necessary  fields  of  the  lesson  learned  form  are  filled  in.  If  the  lesson  is 
received  through  the  Department  of  Energy  Office  of  Environmental  Management 
Occurrence  Reporting  System,  that  system  has  formal  review  components  in  place  so  the 
lesson  does  not  get  reviewed  again. 

The  collection  of  review  systems,  field  office  and  occurrence  reporting  system, 
protect  against  the  release  of  classified  information  and  technical  accuracy  in  accounting 
the  lesson  or  success  story. 

The  Department  of  Energy  Office  of  Environmental  Management  Eessons 
Eearned  Program  has  been  successful.  The  Department  of  Energy  Office  of 
Environmental  Management  Eessons  Eearned  Program  has  evidence  of  cost  savings/cost 
averted  based  on  sharing  of  success  stories  and  lessons  learned. 

The  present  concern  is  for  lessons  learned  in  the  field  to  reach  The  Department  of 
Energy  Office  of  Environmental  Management  Eessons  Eearned  Program.  A  secondary 
concern  is  for  the  lessons  learned  to  reach  the  people  who  can  use  them.i23 

123  McCune,  M.,  Department  of  Energy  Office  of  Environmental  Management  Lessons  Learned 
Program  point  of  contact,  (personal  communication,  e-mail,  July  10,  2002) 
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Based  on  experience  and  success,  some  suggestions  to  future  designers  are 

provided:  124 

1 .  Make  sure  the  database  application  is  supported  and  upgradeable  to  grow 
with  the  Lessons  Learned  System. 

2.  A  good  search  engine  is  important. 

3.  Simplicity  and  ease  of  use  are  the  keys  to  a  Lessons  Learned  System  that 
people  will  use. 

4.  Design  the  system  for  the  ability  to  create  reports  that  trend  the  lessons  in 
the  database  with  ease. 

5.  Design  the  Lessons  Learned  System  so  that  it  is  not  only  pushing 
information  out  but  also  pulling  information  in  (push  pull). 

6.  Provide  a  number  of  different  formats  that  a  lessons  learned  provider  can 
use  to  prepare  a  lessons  learned. 

12,  Federal  Transit  Administration  Lessons  Learned  Systemi25 

The  Federal  Transit  Administration  deals  with  public  transportation.  Public 
transportation  may  include  buses,  rail  vehicles  and  system,  commuter  ferryboats,  trolleys, 
subways,  etc.  The  U.S.  Department  of  Transportation,  through  the  Federal  Transit 
Administration,  provides  financial  and  technical  assistance  to  the  local  transit  systems.  126 

The  purpose  of  the  Federal  Transit  Administration  Lessons  Learned  System  is  to 
share  knowledge  on  the  successes,  the  challenges  (mishaps)  and  applications  of  new 
technology  in  the  building  of  the  United  States’  public  transportation  system.  It  was 
established  in  January  of  1995  and  is  a  developing  Lessons  Learned  System.  127 


124  McCune,  M.,  Department  of  Energy  Office  of  Environmental  Management  Lessons  Learned 
Program  point  of  contact,  (personal  communication,  e-mail,  July  10,  2002) 

125  Nassif,  S,  Federal  Transit  Administration  Lessons  Learned  System  point  of  contact,  (personal 
communication,  telephone  questionnaire,  January  25,  2002) 

126  Federal  Transit  Administration  Web  Page 

127  Nassif,  S,  Federal  Transit  Administration  Lessons  Learned  System  point  of  contact,  (personal 
communication,  telephone,  July  10,  2002) 
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The  usual  method  of  doing  business  is  to  supply  grants  to  munieipalities  for  their 
public  transportation  system.  These  grants  may  be  used  to  build  transit,  for  project 
construction,  for  hardware  such  as  busses  and  to  develop  transit  systems.  The 
municipalities  or  grant  recipients  normally  hire  a  consultant  to  administer  the  project 
financed  by  the  grant.  While  construction  is  taking  place  or  as  the  grant  is  being  used, 
the  consultant  collects  lessons  learned  as  part  of  the  contract. 

Once  a  year,  there  is  a  round  table  discussion  that  includes  the  Federal  Transit 
Administration  Lessons  Learned  System,  the  grant  recipients  and  their  consultants.  Part 
of  the  agenda  is  to  discuss  lessons  learned.  The  lessons  learned  may  involve  financial, 
safety,  design  and  all  other  aspects  of  the  project  sponsored  by  the  grant.  A  report  is 
written  including  the  lessons  learned.  The  report  with  lessons  learned  is  used  by  the 
Federal  Transit  Administration  in  the  planning  and  administering  of  future  grants.  The 
report  with  lessons  learned  is  retained  with  the  National  Transit  Library. 

The  Federal  Transit  Administration  Lessons  Learned  System  has  no  present 
concerns  although  it  is  constantly  seeking  to  improve.  Lessons  learned  are  part  of  each 
grant,  they  are  reviewed  by  policy  making  personnel  at  the  round  table  discussion  and  the 
lessons  learned  are  used  in  future  operations.  The  Federal  Transit  Administration 
Lessons  Learned  System  has  high  management  including  financial  support.  1 28 


13.  International  Space  Station  Lessons  Learned  Databasei29 

The  International  Space  Station  is  a  project  to  build  an  orbiting  laboratory  in 
space  that  will  house  scientists  and  astronauts.  The  International  Space  Station  will  have 
a  mass  of  1,040,000  pounds  and  will  measure  356  feet  across  and  290  feet  long.  It  will 
have  almost  an  acre  of  solar  collectors  and  six  state  of  the  art  laboratories.  It  will  orbit  at 
250  miles.  The  project  is  lead  by  the  United  States  in  partnership  with  Canada,  Japan, 


128  Nassif,  S,  Federal  Transit  Administration  Lessons  Learned  System  point  of  eontaet,  (personal 
eommunieation,  telephone,  July  10,  2002) 

129  Vassberg,  N.,  International  Spaee  Station  Lessons  Learned  Database  point  of  eontaet, 
(personal  eommunieation,  e-mail  questionnaire,  February  8,  2002) 
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Russia,  1 1  nations  of  the  European  Space  Agency  and  Brazil.  Assembly  is  planned  to  be 
complete  in  2004.130 

The  purpose  of  the  International  Space  Station  Lessons  Learned  Database  is  to 
archive  lessons  learned  from  the  International  Space  Station  for  future  NASA  programs 
in  an  easy  to  use/access/categorized  form.  Another  purpose  is  to  document  lessons 
learned  and  measures  taken  to  prevent  recurrence  at  multiple  sites.  This  is  key  for  a 
program  the  size  of  the  International  Space  Station  with  multiple  facilities  in  the  United 
States  and  around  the  world.  It  is  possible  to  learn  the  same  lesson  multiple  times 
without  the  communication  tool  to  transfer  the  learned  knowledge.  The  International 
Space  Station  Lessons  Learned  Database  is  the  tool  used  to  implement  lessons  learned 
and  prevent  recurrence. 

It  has  existed  in  its  present  form  since  1998.  It  began  four  years  earlier  as  a 
collection  of  lessons  obtained  from  the  space  shuttle  docking  with  the  Russian  Mir  Space 
Station.  Those  lessons  were  originally  in  a  spreadsheet.  A  desire  to  disseminate  the 
lessons  for  the  International  Space  Station  was  the  driving  force  behind  the  development 
of  The  International  Space  Station  Lessons  Learned  Database.  131 

Lessons  are  obtained  by  the  organization  that  learns  the  lesson  submitting  the 
lesson  on  a  voluntary  basis.  There  is  no  formal  requirement  to  submit  a  lesson  that  is 
learned  to  the  International  Space  Station  Lessons  Learned  Database.  The  present 
concern,  being  a  passive  collection  system,  is  the  obtaining  of  lessons.  Some  methods  to 
increase  the  submittals  of  lessons  have  been  tried.  One  method  was  a  reward  system.  A 
free  dinner  was  given  to  those  having  lessons  accepted  for  dissemination.  It  stimulated 
lesson  submission  until  the  novelty  wore  off.  1 32 

Once  a  lesson  is  received  by  the  International  Space  Station  Lessons  Learned 
Database,  it  is  processed  in  the  following  way.  Lirst  it  is  categorized  as  belonging  to  one 
of  twelve  possible  technical  discipline  areas.  Lor  each  of  the  twelve  disciplines  there  is  a 

130  International  Space  Station  Web  Page 

1 3 1  Vassberg,  N.,  International  Space  Station  Lessons  Learned  Database  point  of  contact, 

(personal  communication,  telephone,  July  11,  2002) 


132  Ibid. 
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reviewer  who  when  assigned  the  lesson  learned,  determines  first  if  the  lesson  is  a  lesson. 
The  reviewer  also  judges  if  the  right  level  of  detail  is  ineluded  so  that  a  reader  ean 
understand  the  lesson.  The  reviewer  determines  if  the  categorization  is  correct  and  makes 
sure  that  there  is  no  sensitive  or  personal  data  included. 

Once  this  initial  screening  is  completed  by  the  International  Space  Station 
Lessons  Learned  Database,  the  lesson  is  forwarded  to  a  management  board  for  the 
technical  discipline  to  review.  The  management  board  concurs  with  the  initial  screening 
or  makes  changes  or  rejects  the  lesson.  The  two-step  process  assures  the  management 
board  is  not  overloaded  with  lessons  and  is  a  check  on  the  initial  review. 

This  degree  of  formality  on  the  validation  process  is  used  to  insure  that  the 
International  Space  Station  Lessons  Learned  Database  contains  quality  lessons  learned. 
It  is  the  feeling  of  the  International  Space  Station  Lessons  Learned  Database  that  if  the 
database  becomes  cluttered  with  junk  lessons,  the  International  Space  Station  Lessons 
Learned  Database  loses  its  value.  If  its  value  is  reduced,  there  will  be  a  hesitancy  to  use 
the  International  Space  Station  Lessons  Learned  Database.  There  is  a  credibility  that 
exists  when  the  management  board  endorses  a  lesson  and  this  encourages  International 
Space  Station  Lessons  Learned  Database  use. 

The  International  Space  Station  Lessons  Learned  Database  has  been  successful. 
The  corrective  actions  that  have  been  taken  have  demonstrated  a  reduction  in  the 
recurrence  of  problems. 

There  are  several  International  Space  Station  Lessons  Learned  Database 
perceived  consequences  for  disseminating  an  erroneous  or  misleading  lesson  learned. 
One  of  the  consequences  is  that  there  will  be  a  decline  in  usage  of  the  International  Space 
Station  Lessons  Learned  Database.  A  second  consequence  is  that  an  unnecessary  or 
wrong  action  will  be  taken  that  could  effect  safety,  efficiency,  etc.  The  International 
Space  Station  Lessons  Learned  Database  focuses  on  the  root  cause  of  the  lesson.  If  the 
root  cause  is  accurate  then  corrective  actions  should  work. 

The  International  Space  Station  Lessons  Learned  Database  not  only  focuses  on 
obtaining  quality  lessons  but  also  on  distributing  them.  Once  a  lesson  is  approved  by  the 
management  board  for  a  technical  discipline,  the  system  automatically  distributes  it  to 
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predetermined  people  or  groups  who  need  to  see  and  respond  to  the  lessons.  This  assures 
that  the  users  know  that  a  new  lesson  is  in  the  system  that  deals  with  their  area. 

The  dissemination  goes  further.  The  disseminated  lesson  learned  are  tracked  and 
if  a  lesson  learned  is  observed  to  be  idle  in  one  work  station,  that  work  station  is 
reminded  that  appropriate  action  needs  to  be  taken.  1 33 

This  would  be  an  example  of  the  overall  process.  A  site  in  California  learns  a 
lesson  and  submits  the  lesson  to  the  International  Space  Station  Lessons  Learned 
Database.  The  International  Space  Station  Lessons  Learned  Database  reviews  the  lessons 
and  forwards  it  to  the  management  board  of  Test  &  Verification.  The  Management 
Board  of  Test  &  Verification  approves  the  lesson  learned.  The  lesson  learned  is  then  sent 
by  e-mail  to  the  Test  &  Verification  groups  at  all  program  sites.  Key  individuals  are 
required  to  respond  to  the  International  Space  Station  Lessons  Learned  Database  to 
document  if/how  the  lesson  applies  to  them  and  what  they  have  done  as  a  result  of  the 
lesson.  The  database  captures  these  responses  as  part  of  the  original  lesson.  This  closed 
loop  assures  that  the  International  Space  Station  is  learning  from  its  lessons. 

The  development  of  The  International  Space  Station  Lessons  Learned  Database 
continues.  Capitalizing  on  their  in  house  expertise,  the  software  involved  in 
dissemination  of  lessons  learned  is  state  of  the  art.  Video  and  audio  are  now  included  in 
the  dissemination  of  lessons  learned  bringing  with  them  all  their  advantages.  The 
resource  drain  of  The  International  Space  Station  Lessons  Learned  Database  is  not  great 
as  tasks  are  spread  throughout  the  organization  thus  creating  no  great  burden  for  a  few. 

The  key  to  The  International  Space  Station  Lessons  Learned  Database  success, 
aside  from  design  and  processes,  is  management  support  at  all  levels,  particularly  upper 
management.  The  positive  effects  on  lessons  learned  submission  activity  as  a  result  of  a 
few  passing  words  regarding  lessons  learned  importance  by  high  profile  managers  to 
lower  managers  and  workers  is  noticeable. 1 34 


133  Vassberg,  N.,  International  Space  Station  Lessons  Learned  Database  point  of  contact, 
(personal  communication,  telephone,  July  11,  2002) 


134  Ibid. 


76 


14,  Mine  Action  Information  Center  Lessons  Learned  Databasei35 


The  Mine  Action  Information  Center  is  an  established  Center  of  Excellence  by  the 
Department  of  Defense  at  James  Madison  University.  Its  mandate  is  to  collect,  process, 
analyze  and  disseminate  information  relevant  to  humanitarian  mine  action  clearance, 
victim  assistance,  community  risk  reduction,  refugee  resettlement  and  other  land  mine 
related  issues.  Its  partners  include  The  Department  of  State,  the  Department  of  Defense, 
the  Slovenian  International  Trust  Fund,  the  Canadian  government  and  The  Geneva 
International  Center  for  Humanitarian  Demining. 

The  purpose  of  the  Mine  Action  Information  Center  Lessons  Learned  Database  is 
to  capture  lessons  learned  from  humanitarian  demining  operations.  It  is  designed  to  serve 
the  entire  mine  action  community  by  providing  a  method  and  forum  for  distributing 
experiences  and  methodologies  that  may  be  of  benefit  to  others.  It  began  operation  in  the 
spring  of  2001. 

Lessons  are  obtained  from  operators  who  enter  them  into  the  system  after  a 
deployment.  The  Mine  Action  Information  Center  Lessons  Learned  Database  Internet 
website  also  allows  lessons  learned  to  be  entered  by  anyone  who  will  register  into  the 
system. 

There  is  no  validation  of  lessons.  Lessons  are  accepted  as  is.  The  reason  there  is 
no  validation  process  is  to  encourage  the  widest  scope  and  amount  of  input.  The  Mine 
Action  Information  Center  Lessons  Learned  Database  Intranet  website  also  allows 
comments  to  be  made  with  regard  to  lessons  posted  so  in  a  sense,  the  system  is  self¬ 
policing.  The  validity  of  the  lesson  learned  can  be  judged  by  comments  entered  in 
reference  to  the  lesson. 

The  Mine  Action  Information  Center  Lessons  Learned  Database  does  not  consider 
itself  to  be  fully  successful  yet.  It  cites  a  limited  amount  of  input  information.  To 
improve  upon  this,  the  Mine  Action  Information  Center  Lessons  Learned  Database  is 
conducting  an  outreach-marketing  plan. 


135  Barlow,  D.,  Mine  Action  Information  Center  Lessons  Learned  Database  point  of  contact, 
(personal  communication,  e-mail  questionnaire.  May  22,  2002) 
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The  consequences  of  disseminating  an  erroneous  or  misleading  lesson  learned  is 
not  of  as  great  of  concern  as  there  being  limited  sharing  of  demining  lessons  learned.  In 
other  words,  at  this  point  in  the  Mine  Action  Information  Center  Lessons  Learned 
Database  development,  the  Mine  Action  Information  Center  Lessons  Learned  Database 
would  prefer  quantity  of  lessons  over  quality  and  allow  a  self-policing  to  establish  the 
quality. 

There  is  a  disclaimer  associated  with  the  Mine  Action  Information  Center  Lessons 
Learned  Database  that  states  that  messages  are  not  edited  for  content  and  opinions  are 
those  of  the  users  posting  information  and  are  not  attributable  to  the  Mine  Action 
Information  Center  Lessons  Learned  Database  or  its  partners.  136 

Recently,  there  has  been  a  revision  to  operating  procedure.  Lessons  are  now  also 
sought  from  open  literature  and  entered  into  the  database  by  Mine  Action  Information 
Center  Lessons  Learned  Database  personnel.  1 37 


15,  Electric  Boat  Corporate  Lessons  Learned  Databasei38 

With  more  than  a  century  of  experience.  Electric  Boat  has  established  standards 
of  excellence  in  the  design,  construction  and  lifecycle  support  of  submarines  for  the  U.S. 
Navy.  Primary  operations  are  the  shipyard  in  Groton,  CT,  and  the  automated  hull- 
fabrication  and  outfitting  facility  in  Quonset  Point,  RI,  with  a  current  workforce  of  nearly 
9,000  employees. 139 

As  a  good  business  practice.  Electric  Boat  has  established  the  Electric  Boat 
Corporate  Eessons  Eearned  Database.  Eesson  Eearned  Systems  have  existed  at  Electric 
Boat  for  some  time.  These  systems  were  local  in  nature.  Each  design  project  included  a 
lessons  learned  system  and  lessons  learned  systems  were  also  a  part  of  functional  groups 

136  Mine  Action  Information  Center  Lessons  Learned  Database  Web  Page 

137  Barlow,  D.,  Mine  Action  Information  Center  Lessons  Learned  Database  point  of  contact, 
(personal  communication,  e-mail,  July  11,  2002) 

138  Thaxton,  D.,  Electric  Boat  Corporate  Lessons  Learned  Database  point  of  contact,  (personal 
communication,  questionnaire,  June  22,  2002) 

139  General  Dynamics  Electric  Boat  Web  Page 
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such  as  testing  or  nuclear  engineering.  These  systems  were  independently  operated. 
Some  were  paper  and  some  were  electronic.  The  Electric  Boat  Corporate  Lessons 
Learned  Database  was  established  to  umbrella  all  of  the  local  lessons  learned  systems.  It 
is  an  Intranet  website. 

The  purpose  of  the  Electrie  Boat  Corporate  Lessons  Learned  Database  is  to 
provide  global  lessons  learned  during  design,  manufacture,  test  and  operation  of  Navy 
ships  and  land  based  prototypes  to  the  Electric  Boat  community.  This  database  does  not 
supereede  existing  department  or  project  lesson  learned  fdes,  but  provides  an  area  where 
eritieal  lessons,  both  suceesses  and  failures,  are  easily  accessible  by  the  larger 
community. 

Lessons  are  obtained  from  the  local  lessons  learned  systems.  The  management  of 
the  loeal  lessons  learned  systems  judges  that  a  lesson  learned  existing  in  the  local  system 
has  value  beyond  the  local  level.  The  lesson  learned  has  value  to  the  larger  Electric  Boat 
community.  The  loeal  management  then  enters  the  lesson  learned  into  an  electronic 
submittal  form  of  the  Eleetric  Boat  Corporate  Lessons  Learned  Database  intranet  website. 
Lessons  learned  are  also  obtained  by  seanning  a  lessons  learned  database  of  a  wide  area 
network  serving  a  select  group  of  organizations.  Another  souree  of  lessons  is  by  the 
implementation  of  shipyard  proeedures  governing  work  reviews. 

Lessons  learned  of  the  Eleetrie  Boat  Corporate  Lessons  Learned  Database  are 
validated  by  two  levels  of  review.  The  first  level  is  by  the  submitting  souree.  The 
lessons  are  judged  to  be  accurate  and  global  by  local  lessons  learned  managers,  wide  area 
network  publishers  or  management  presiding  over  work  reviews.  The  seeond  validation 
is  by  an  Electric  Boat  Corporate  Lessons  Learned  Database  review  board  consisting  of 
five  managers  representing  major  disciplines  at  Eleetric  Boat.  These  diseiplines  are 
engineering,  operations,  test,  quality  eontrol  and  radiation  eontrol.  The  eriteria  for 
validation  includes  that  lessons  have  properly  undergone  Navy  root  eause  analysis,  are 
globally  applieable  and  are  written  to  a  standard  appropriate  for  dissemination. 

The  Electrie  Boat  Corporate  Lessons  Learned  Database  has  been  somewhat 
suecessful.  There  is  evidence  that  the  database  is  being  accessed  by  the  Electric  Boat 
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community  for  lessons  learned.  There  may  be  a  need  to  increase  the  population  of 
lessons  learned  in  the  Electrie  Boat  Corporate  Lessons  Learned  Database. 

There  is  an  Eleetric  Boat  requirement  that  new  design  projects  review  the  lessons 
learned  from  past  design  projects.  The  development  of  database  teehnology  with 
inereased  speed  in  searching  for  specific  subjects  with  user  friendliness  has  made  a  global 
lessons  learned  system  feasible  at  Eleetrie  Boat. 


C.  ORGANIZATION  OF  EXISTING  LESSONS  LEARNED  DATA 

The  data  is  organized  to  support  the  methodology  of  analysis.  The  phrase  existing 
Lessons  Learned  System  represents  the  sample  of  Lessons  Learned  System  in  Section  B 
and  is  not  meant  to  represent  all  existing  Lessons  Learned  Systems. 
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A  key  code  is  established  for  each  Lessons  Learned  System  as  indicated  in  Table 
IV- 1  below: 


Table  IV-1  Key  Code  of  Existing  Lessons  Learned  Systems 

Key 

Code 

Lessons  Learned  System 

1 

Canadian  Army  Lessons  Learned  Centre 

2 

The  United  Nations  Peacekeeping  Best  Practices  Unit  (formerly  The  Lessons 
Learned  Unit  of  the  Department  of  Peacekeeping  Operations) 

3 

American  Industrial  Hygiene  Association  (AIHA)  Health  &  Safety 

Committee 

4 

U.S.  Army  Center  for  Engineer  Lessons  Learned 

5 

Army  Medical  Department  Lessons  Learned 

6 

Coast  Guard  -  Standard  After  Action  Information  and  Lessons  Learned 

System 

7 

Best  Manufacturing  Practices  Program 

8 

Department  of  Energy  Project  Hanford  Eessons  Eearned  System 

9 

BNEE  Incorporated  Eessons  Eearned  System 

10 

Department  of  Energy  Headquarters  Eessons  Eearned  System 

11 

Department  of  Energy  Office  of  Environmental  Management  Eessons 

Eearned  Program 

12 

Eederal  Transit  Administration  Eessons  Eearned  System 

13 

International  Space  Station  Eessons  Eearned  Database 

14 

Mine  Action  Information  Center  Eessons  Learned  Database 

15 

Electric  Boat  Corporate  Lessons  Learned  Database 
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Table  IV -2  below  lists  the  data  from  whieh  a  Handling  Variable  Set  ean  be 
determined. 


Table  IV-2  Handling  Methods  of  Existing  Lessons  Learned  Systems 

Key 

Code 

Handling  Methods 

1 

written  by  source  on  standard  form,  reviewed  by  supervisor,  analyzed  and  root 
cause  determined  centrally,  suggestions  forwarded,  feedback  to  initiator 

2 

reports  generated  at  site,  edited  by  group  of  experts  including  users  and  policy 
makers  centrally,  final  report  for  dissemination 

3 

lessons  received,  reviewed  by  group,  edited  to  make  generic,  published 

4 

observations  forwarded  (mild  requirement),  reviewed  by  experts  centrally, 
perform  tests  to  verify 

5 

observations  forwarded  (mild  requirement),  reviewed  by  experts  and  solutions 
developed  centrally,  feedback  on  solutions 

6 

written  by  source  and  forwarded  (mild  requirement),  command  review, 
reviewed  centrally,  edited  as  needed,  published 

7 

included  in  write-up  as  format  of  implementation  review,  no  review 

8 

actively  search  for  lessons,  filter  and  reviewed  by  experts  centrally 

9 

actively  search  for  lessons,  developed  centrally,  validated  by  checking  results 
and  monitoring  for  the  reoccurrence  of  event 

10 

actively  develop  by  critiques  and  work  procedures  requirement,  subject  matter 
experts  to  develop 

11 

voluntary  submission,  actively  search  (secondary),  reviewed  centrally  for 
editorial  and  public  relations 

12 

requirement  of  contract  to  include,  general  review  by  policy  makers,  publish  as 
report 

13 

received  from  volunteers,  categorize  and  review,  higher  review  and 
acceptance,  selectively  disseminated  with  an  action  to  respond  requirement 

14 

written  by  source/voluntarily  submitted,  no  processing,  allow  comments  to  be 
posted  against 

15 

submitted  voluntary  from  local,  reviewed  by  local,  reviewed  centrally, 
published 
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Table  IV-3  below  identifies  the  organizational  aspects  of  the  Lessons  Learned 
System  that  are  applicable  to  the  methodology.  These  are  not  the  organizational 
characteristics  that  are  referred  to  in  Chapter  IT  Included  is  the  goal  or  purpose  of  the 
organization,  the  development  stage  of  the  Lessons  Learned  System  and  the  present 
concern.  The  present  concern  is  either  population  of  lessons,  quality  of  lessons,  use  of 
lessons  by  the  organization. 


Table  IV-3  Organizational  Aspects  of  Existing  Lessons  Learned  Systems 

Key 

Code 

Goal  or  Purpose 

Development 

Stage 

Present 

Concern 

1 

improve  operational  capability  and  efficiency 

mature 

implementi 
ng  lessons 

2 

recommend  actions  for  future  operations 

developing 

implementi 
ng  lessons 

3 

knowledge  sharing  (collect  and  supply  mishap 
information) 

developing/ 

mature 

receiving 

lessons 

4 

improve  engineering  performance 

developing 

receiving 

lessons 

5 

improve  methods  of  operation,  medical 

new/ 

developing 

receiving 

lessons 

6 

enhance  unit  preparedness 

new/ 

developing 

receiving 

lessons 

7 

provide  awareness  concerning  implementation 

8 

improve  efficiency,  operations  and  prevent 
accidents  (safety) 

mature 

implementi 
ng  lessons 

9 

promote  efficiency  and  safety 

developing/ 

mature 

implementi 
ng  lessons 

10 

improve  efficiency  and  prevent  accidents 
(safety) 

mature 

implementi 
ng  lessons 

11 

knowledge  sharing  of  operations  and  safety 
(environmental) 

developing/ 

mature 

receiving 

lessons 

12 

knowledge  sharing  of  operations 

developing/ 

mature 

none 

13 

improve  efficiency  of  operations  and 
knowledge  sharing 

developing/ 

mature 

receiving 

lessons 

14 

knowledge  sharing  of  operations  and  safety 

new/ 

developing 

receiving 

lessons 

15 

knowledge  sharing  to  improve  efficiency  of 
operations 

new/ 

developing 

receiving 

lessons 
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Table  IV-4  identifies  the  operational  oharacteristies  of  the  Lessons  Learned 
Systems.  These  include  identification  of  the  Formality  characteristic  (formal  or  ad  hoc), 
the  Locus  characteristic  (centralized  or  distributed),  the  Process  Relation  characteristic 
(embedded  or  standalone)  and  the  Acquisition  characteristic  (active  or  passive). 


Table  IV-4  Operational  Characteristics  of  Existing  Lessons  Learned  Systems 

Key 

Code 

Formality 

Locus 

Process  Relation 

Acquisition 

1 

formal 

centralized 

combination 

combination 

2 

formal 

centralized 

embedded 

active 

3 

ad  hoc 

centralized 

standalone 

passive 

4 

formal 

centralized 

combination 

combination 

5 

formal 

centralized 

combination 

combination 

6 

formal 

centralized 

combination 

passive 

7 

ad  hoc 

centralized 

standalone 

passive 

8 

formal 

centralized 

combination 

combination 

9 

formal 

centralized 

combination 

combination 

10 

formal 

distributed 

combination 

combination 

11 

formal 

distributed 

combination 

combination 

12 

ad  hoc 

centralized 

embedded 

active 

13 

formal 

distributed 

standalone 

passive 

14 

ad  hoc 

centralized 

standalone 

passive 

15 

formal 

distributed 

combination 

passive 
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Table  IV-5  below  identifies  the  lesson  characteristics  of  the  Lessons  Learned 
Systems.  These  include  the  Content  characteristic  (pure  or  hybrid)  and  the  Process  Type 
characteristic  (technical,  administrative  or  planning).  It  should  be  understood  that  most 
of  the  Lessons  Learned  Systems  have  some  part  of  all  the  qualitative  values.  Table  IV-5 
represents  a  best  effort  predominant  value  based  on  the  data  on  hand. 


Table  IV-5  Lesson  Characteristics  of  Existing  Lessons  Learned  Systems 

Key 

Code 

Content 

Process  Type 

1 

pure 

planning/technical 

2 

pure 

planning 

3 

pure 

technical 

4 

pure 

technical 

5 

pure 

technical/planning 

6 

hybrid 

technical/ administrative 

7 

pure 

technical 

8 

hybrid 

technical 

9 

hybrid 

technical 

10 

hybrid 

technical 

11 

hybrid 

technical 

12 

pure 

planning/technical 

13 

pure 

technical 

14 

pure 

technical 

15 

pure 

technical 
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Table  IV-6  below  identifies  the  organizational  oharaeteristies  of  the  Lessons 
Learned  Systems.  These  inelude  the  Interpretive  Context  eharaeteristie  (high,  medium  or 
low)  and  the  Type  eharaeteristie  (adaptable,  rigid).  Table  IV-6  represents  a  best  effort 
predominant  value  based  on  the  data  on  hand. 


Table  IV-6  Organizational  Characteristics  of  Existing  Lessons  Learned  Systems 

Key 

Code 

Interpretive  Context 

Type 

1 

medium 

rigid 

2 

low 

rigid 

3 

medium 

adaptable 

4 

medium 

rigid 

5 

high 

adaptable 

6 

medium 

adaptable 

7 

medium 

adaptable 

8 

medium 

adaptable 

9 

medium 

adaptable 

10 

medium 

adaptable 

11 

medium 

adaptable 

12 

medium 

adaptable 

13 

medium 

adaptable 

14 

medium 

adaptable 

15 

medium 

adaptable 

86 


Table  IV-7  below  identifies  the  organizational  characteristics  of  the  Lessons 
Learned  Systems.  These  include  the  Resources  available  (high,  medium  or  low)  and 
Responsibility  (high,  medium  or  low).  Responsibility  is  the  level  of  responsibility  that  a 
Lessons  Learned  System  holds  for  the  accuracy  of  Lessons  Learned  disseminated.  Table 
IV-7  represents  a  best  effort  based  on  the  Organization  and  disclaimer,  if  existing. 


Table  IV-7  Other  Organizational  Characteristics  of  Existing  Lessons  Learned 

Systems 

Key 

Code 

Resources 

Responsibility 

1 

medium/me  dium  1 40 

medium 

2 

medium/high 

low 

3 

low 

low 

4 

medium/high 

high 

5 

medium/high 

high 

6 

low/medium 

medium 

7 

low 

low 

8 

medium/high 

medium 

9 

medium/high 

medium 

10 

medium/high 

medium 

11 

medium/high 

low 

12 

medium 

low 

13 

medium/high 

medium  (self  imposed  high) 

14 

low 

low 

15 

medium 

medium 

140  resources  of  Lessons  Learned  System/resources  of  parent  organization 
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Table  IV-8  below  identifies  key  statements  made  by  the  Lessons  Learned 
Systems.  Table  IV-8  does  not  eontain  all  key  statements  and  referenee  should  be  made  to 
seetion  B  for  those  statements  and  for  eontext  of  the  statements  below. 


Table  IV-8  Key  Statements  of  Existing  Lessons  Learned  Systems 

Key 

Code 

Statement 

1 

biggest  ehallenge  is  that  ehange  is  implemented,  the  importanee  of  ehain  of 
eommand  in  the  proeess 

2 

poliey  makers  are  involved  in  the  development  of  lessons  learned  report 

3 

most  submissions  of  lessons  have  been  those  sought  out 

4 

lessons  gathering  is  mostly  passive  and  there  is  eoncern  that  there  are  lessons 
learned  that  exist  in  the  field  that  are  not  being  obtained 

5 

lessons  gathering  relies  on  input  from  field,  viewed  by  field  as  low  priority 
(non  management  support),  missed  opportunities 

6 

concern  of  lessons  being  obtained,  submitted  voluntarily 

7 

included  as  part  of  review 

8 

importance  of  management  support  at  all  levels,  make  the  system  fit  within 
the  tools  your  workers  use  every  day,  lack  of  quality  lessons  equates  to  non 
use 

9 

quality  of  lessons  equals  continued  usage,  if  safety  involved,  override  time 
consuming  review  process,  disseminate  lesson  with  disclaimer,  then  follow 
up 

10 

lesson  reviewed  by  subject  matter  expert  for  quality,  requirement  of  an 
operational  procedure  is  documenting  lessons  learned 

11 

simplicity  and  ease  of  use  equals  usage  of  system 

12 

lessons  learned  contractual  requirement,  collectors  and  policy  makers  writing 
lesson  learned  report  together 

13 

proactive  dissemination  and  implementation,  organizational  wide 
management  support,  technical  expert  reviews,  high  management 
endorsement,  high  quality  lessons  equal  usage 

14 

limited  lessons  input ,  action  to  counter  is  outreach  marketing  plan  and  active 
search 

15 

passive  system  equals  lesson  population  worries,  usage  a  requirement  of  new 
products,  user  friendly  information  technology  makes  system  feasible 
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D,  CONCLUSION 


The  chapter  contains  the  data  collected  about  existing  Lessons  Learned  Systems. 
It  organized  the  data  in  a  form  that  supports  the  methodology  of  Chapter  III  as 
implemented  in  Chapter  V.  Section  B  contained  write-ups  on  the  existing  Lessons 
Learned  Systems  based  on  a  questionnaire  and  e-mail  and  telephone  follow-up.  Section 
C  organized  the  data  in  tables. 
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V.  ANALYSIS 


A.  INTRODUCTION 

This  chapter  analyzes  the  data  and  answers  the  primary  research  question.  The 
primary  researeh  question  of  this  thesis  is:  How  may  a  Lessons  Learned  System  be 
eharacterized  according  to  the  handling  of  lessons  processing,  and  how  may  such  a 
characterization  be  applied  to  Lessons  Learned  System  design  or  architeeture? 

The  data  is  analyzed  aceording  to  the  methods  outlined  in  Chapter  III. 
Information  on  existing  Lessons  Learned  Systems  was  presented  in  Chapter  IV.  The  data 
was  also  organized  in  tables. 

The  definition  for  lesson  in  Chapter  II  was  not  consistent.  The  first  task  is  to 
establish  a  definition  for  lesson  so  that  the  level  of  treatment  identified  in  Chapter  IV  ean 
be  eollected.  The  definition  for  lesson  and  thus  the  starting  point  for  handling  is 
established  in  Seetion  B. 

The  next  task  is  to  determine  the  Handling  Variable  Set.  This  is  done  by 
colleeting  all  handling  methods  and  creating  a  “union”  set.  The  “union”  set  or  the 
Handling  Variable  Set  is  a  set  of  handling  methods  sueh  that  any  element  of  the  set  can 
be  found  in  one  of  the  Lessons  Learned  Systems  of  Chapter  IV.  The  development  of  the 
Handling  Variable  Set  is  accomplished  in  Section  C. 

The  final  task  will  be  to  identify  the  cause  and  effeet  of  handling  methods  and 
other  influences,  as  viewed  by  the  existing  Lessons  Learned  Systems  of  Chapter  IV,  on 
the  three  tasks  of  a  Lessons  Learned  System.  The  three  tasks  are  eollecting  lessons, 
providing  a  quality  lesson  learned  for  dissemination  and  dissemination  where 
implementation  is  the  goal.  This  is  done  in  Section  D. 
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B.  THE  SCOPE  OF  THE  HANDLING  CHARACTERISTIC 

The  definition  of  lesson  is  not  entirely  consistent  between  the  Army,  the 
Department  of  Energy  and  a  representative  from  the  American  Association  for  Artificial 
Intelligence.  By  strict  definition,  a  lesson  is  something  learned  by  study  or  experience.  141 

Army  Regulation  11-33  does  not  specifically  define  a  lesson  but  does  define  a 
lesson  learned.  A  lesson  learned  is  validated  knowledge  and  experience  derived  from 
observations  and  historical  study  of  military  training,  exercises,  and  combat 
operations.  142  At  first  glance,  it  would  appear  that  the  phrase  lessons  learned  is 
redundant  but  there  is  a  distinction  that  should  be  recognized.  The  use  of  “learned  by 
study  or  experience”  in  the  strict  definition  should  be  viewed  as  the  consequence  of  an 
action.  To  expand,  some  action  takes  place  and  as  a  result  there  is  some  consequence  of 
that  action.  Recognizing  the  connection  between  the  action  and  its  consequence  is  a 
lesson.  The  consequence  may  need  to  be  refined  in  order  to  be  of  use  in  organizational 
learning.  That  is,  it  may  need  to  be  reduced  to  a  root  cause  or  determined  if  it  is 
applicable  on  a  more  wide  scale.  This  process  would  be  the  validation  part  of  the  Army 
definition  of  a  lesson  learned. 

The  reason  for  the  effort  in  fine-tuning  the  definition  of  lesson  is  to  identify  more 
clearly  where  the  characteristic  of  Handling  begins.  By  interpreting  a  lesson  as  an 
experienced  action  and  recognized  consequence,  the  Handling  characteristic  can  begin 
from  the  actions  performed  on  the  lesson  from  that  point.  This  position  may  be 
somewhat  awkward  with  regards  to  the  characteristic  Acquisition.  It  would  seem  logical 
that  Acquisition  would  occur  before  Handling  but  this  is  not  necessarily  the  case.  A  few 
of  the  existing  Lessons  Learned  Systems  receive  Lessons  Learned  that  have  already 
undergone  some  form  of  Handling.  Lor  example,  the  Coast  Guard  -  Standard  After 
Action  Information  and  Lessons  Learned  System  acquires  Lessons  Learned  after  they 
have  been  command  approved.  To  define  Handling  strictly  as  actions  of  a  Lessons 

141  Webster’s  Seventh  New  Collegiate  Dictionary  (1972). 

Army  Regulation  11-33,  Glossary 
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Learned  System  after  Aequisition  would  remove  the  Handling  events  that  should  be 
eonsidered,  sueh  as  the  eommand  approval  of  Lessons  Learned  in  the  Coast  Guard  - 
Standard  After  Aetion  Information  and  Lessons  Learned  System. 

Army  Regulation  11-33  defines  an  observation  as  raw  information  from  any 
souree  whieh  has  not  been  refined  through  analysis.  143  Therefore,  using  Army  language, 
a  definition  of  Handling  eould  be  revised  from  “Handling  refers  to  the  level  of  treatment 
given  a  lesson  after  it  has  been  generated”  to  “Handling  refers  to  the  level  of  treatment 
given  an  observation  after  it  has  been  reeognized.” 

The  Department  of  Energy  Standard  DOE-STD-7501-99  Deeember  1999  does  not 
define  lesson  but  it  does  define  a  Lesson  Learned. 

A  Lesson  Learned  is  a  good  work  praetiee  or  innovative  approaeh  that  is  eaptured 

and  shared  to  promote  repeat  applieation.  A  lesson  learned  may  also  be  an  adverse 

work  practice  or  experience  that  is  captured  and  shared  to  avoid  recurrence.  144 

Defining  the  generation  of  a  lesson  as  a  recognized  action  with  its  consequence 
and  Handling  as  the  actions  that  occur  from  that  point  on  does  not  contradict  the 
Department  of  Energy  Standard  DOE-STD-7501-99  December  1999  definition  for  a 
Eesson  Eeamed. 

It  would  not  be  consistent  with  the  definition  for  a  lesson  introduced  by  Aha 
(2000).  That  definition  for  lesson  is  a  validated  record  extracted  from  a  (positive  or 
failure)  experience  with  a  previous  decision  process  that  others  in  an  organization  can 
reuse  to  reinforce  a  positive  result  and/or  avoid  a  failure.  145  This  definition  implies  that 
some  Handling  has  already  occurred.  There  is  the  statement  that  the  record  has  been 
validated  and  that  others  in  the  organization  can  reuse  the  lesson.  To  use  this  definition 
of  a  lesson  in  the  definition  of  Handling  would  reduce  the  scope  of  what  is  considered  the 
Handling  level  of  treatment  to  those  actions  concerned  with  dissemination,  such  as 
editorial  preparation  and  appropriate  distribution. 


Army  Regulation  11-33,  Glossary 
144  DOE-STD-7501-99. 


145  Aha,  D.  W.  (2000). 
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The  subject  of  dissemination  raises  another  question  that  should  be  addressed. 
When  is  the  Handling  aspect  considered  to  be  finished?  Does  it  end  when  the  lesson 
learned  is  disseminated  or  does  handling  include  the  actions  after  dissemination?  Some 
existing  Lessons  Learned  Systems  perform  actions  after  a  Lesson  Learned  is 
disseminated.  For  example,  the  International  Space  Station  Lessons  Learned  Database 
requires  that  groups  who  are  sent  a  Lesson  Learned  respond  as  to  how  the  Lesson 
Learned  affects  them  and  if  so,  what  actions  are  being  implemented  to  capitalize  on  the 
Lesson  Learned. 

To  include  the  post-dissemination  actions  into  the  characteristic  of  Handling  may 
expand  the  Handling  characteristic  to  such  a  point  that  it  would  be  difficult  to  provide  a 
simple  qualitative  value  to  describe  it.  There  is  also  a  characteristic  of  Dissemination.  It 
has  qualitative  values  of  active  and  passive.  The  post-dissemination  actions  could  be 
included  within  the  qualitative  value  of  active. 

Because  of  the  two  considerations  above,  the  handling  of  lessons  learned  after 
dissemination  will  not  be  included  within  the  Handling  characteristic. 

The  scope  of  the  Handling  characteristic  includes  the  level  of  treatment  given  a 
lesson  from  when  a  lesson  is  generated,  where  a  lesson  is  an  action/consequence 
experience,  to  the  time  when  the  lesson  learned  is  disseminated. 
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C.  THE  HANDLING  VARIABLE  SET 

The  Handling  Variable  Set  is  determined  by  examination  of  Table  IV-2. 
Handling  methods  are  ehosen  and  entered  into  Table  V-1.  The  eriteria  for  being  ehosen 
is  that  the  method  is  an  aetion  performed  by  the  Lessons  Learned  System  on  a  lesson, 
where  a  lesson  is  an  experienee  realized  from  an  aetion  and  the  aetion  performed  by  the 
Lessons  Learned  System  oceurs  from  the  time  a  lesson  is  realized  to  when  it  is 
disseminated.  All  the  methods  of  Table  IV-2  that  meet  the  eriteria  are  represented  in 
Table  V-l.  To  reduee  the  size  of  Table  V-1,  some  methods  are  grouped  together  due  to 
their  similarity. 

Aetively  searehing  for  lessons  was  not  ineluded  in  Table  IV-2.  The  reason  being 
that  this  aetion  belongs  more  to  the  Aequisition  eharaeteristie  than  to  the  Handling 
charaeteristie.  It  is  dependent  on  the  existence  of  another  Lessons  Learned  System  and  is 
not  an  action  of  an  independent  Lessons  Learned  System. 

Each  action  of  the  Lessons  Learned  Systems  in  Table  V-1  is  assigned  a  variable 
number.  This  number  represents  its  value  place  of  the  Handling  Variable  Set.  For 
example,  variable  number  seven  represents  the  ones  place  and  variable  number  six 
represents  the  tens  place. 
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Table  V-1  Handling  Variable  Set 

Variable 

Number 

Description 

1 

co-written  at  source  by  Lessons  Learned  System  personnel 

2 

reviewed  at  source  by  supervisory  personnel  or  command  approved  or 
preliminary  level  of  review 

3 

reviewed  centrally  for  technical  adequacy  by  experts 

4 

verified  by  test,  or  feedback  once  disseminated 

5 

reviewed  centrally  for  editorial  adequacy  (root  cause,  generic, 
background,  public  relations,  relevancy,  etc.) 

6 

reviewed  centrally  or  otherwise  by  potential  users/policy  implementers 
prior  to  dissemination 

7 

identifies  target  for  dissemination,  may  also  require  response 

A  Handling  Variable  Set  ean  then  be  defined  for  a  Lessons  Learned  System.  The 
set  eonsists  of  seven  binomial  numbers  where  a  one  represents  that  the  Lessons  Learned 
Handling  eharaeteristie  ineludes  the  deseription  and  a  zero  indieates  that  it  does  not.  A 
dash  is  inserted  after  the  first  and  fifth  digit  for  reasons  that  are  apparent  later.  For 
example,  a  Lessons  Learned  System  with  a  Handling  set  of  0-0001-00  would  be  a 
Lessons  Learned  System  that  passively  aeeepts  lessons  learned,  reviews  editorially,  then 
disseminates  in  a  general  fashion. 
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Table  V-2  below  demonstrates  how  a  Lessons  Learned  System  may  be 
eharaeterized  aeeording  to  the  handling  of  lessons  proeessing,  the  first  part  of  the  primary 
researeh  question.  The  Key  Code  numbers  are  from  Table  IV- 1. 


Table  V-2  Handling  Variable  Set  for  Existing  Lessons  Learned  Systems 

Key  Code 

Handling  Variable  Set 

1 

0-1001-01 

2 

1-0101-10 

3 

0-0001-00 

4 

0-1111-01 

5 

0-1111-01 

6 

0-1001-00 

7 

0-0001-00 

8 

0-1101-00 

9 

0-0111-00 

10 

0-1101-00 

11 

0-1001-00 

12 

1-1001-10146 

13 

0-1111-11 

14 

0-0000-00 

15 

0-1001-10 

146  In  analyzing  this  Lessons  Learned  System,  the  hired  consultants  are  considered  as  Lessons 
Learned  System  agents  and  their  actions  are  the  actions  of  the  Lessons  Learned  System 
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D,  ASPECTS  OF  LESSONS  LEARNED  SYSTEM  DESIGN 


In  Chapter  II,  three  tasks  or  requirements  were  identified  for  a  Lessons  Learned 
System.  These  were  reeeiving  lessons,  insuring  that  lessons  learned  are  of  high  quality 
and  dissemination  leading  to  implementation. 

1.  Receiving  Lessons 

A  number  of  existing  Lessons  Learned  Systems  identified  a  coneern  about 
reeeiving  lessons  learned.  From  Table  IV-3,  these  were  Key  Codes  3,  4,  5,  6,  11,  13,  14 
&  15. 

These  Key  Codes  were  at  varying  stages  of  development  and  there  was  no 
definite  correlation  between  being  new  and  a  concern  about  receiving  lessons  learned. 

There  was  a  connection  between  the  Acquisition  characteristic  and  the  concern 
for  receiving  lessons  learned.  From  Table  IV-4,  Key  Codes  3,  4,  5,  6,  11,  13,  14  &  15 
had  an  Acquisition  characteristic  qualitative  value  of  passive  or  combination.  Those 
having  a  qualitative  value  of  combination.  Key  Codes  4  &  5,  identified  that  their  active 
portion,  although  existing,  was  weak.  There  was  a  direct  correlation  between  a  concern 
for  receiving  lessons  learned  and  an  Acquisition  characteristic  qualitative  value  of 
passive.  This  finding  supports  the  literature  review  statement  “information,  you  have  to 
get  it  yourself”  See  Table  II-2. 

Some  of  the  Key  Codes  that  had  a  concern  about  receiving  lessons  learned  had  an 
Operational  Process  Relation  characteristic  qualitative  value  of  embedded  or 
combination.  These  were  Key  Codes  4,  5,  6,  11  &  15.  There  were  two  reasons  why  an 
embedded  qualitative  value  of  the  Process  Relation  characteristic  did  not  guarantee  a 
quantity  of  lessons  learned.  There  were  two  cases.  Key  Codes  11  &  15  where  the 
Lessons  Learned  System  was  part  of  a  distributed  system  and  there  was  a  voluntary  or 
passive  requirement  to  submit  the  lessons  learned.  The  second  case.  Key  Codes  4,  5  &  6 
depend  on  lower  management  enforcement  on  completing  lessons  learned  paperwork. 
These  findings  support  the  literature  review  statement  “performing  work  and  asked  to 
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record  lessons  while  doing  so  is  unlikely  to  be  successful”  and  the  importance  of 
“management  support  and  involvement.”  See  Table  II-2. 

Upper  management  support  by  vocalization  was  also  mentioned  as  having  a 
positive  effect  on  the  generation  of  lessons  learned  as  was  incentives  to  contribute  (Key 
Code  13).  The  positive  effects  of  these  were  cited  as  not  being  permanent  effects.  Again 
this  supports  the  literature  position  of  the  importance  of  management  support. 

Key  Codes  1,  2,  8,  9,  10  &  12  did  not  have  a  concern  about  receiving  lessons 
learned.  The  Lessons  Learned  System  of  Key  Code  7  is  a  secondary  concern  and 
therefore  Key  Code  7  cannot  relate  experience. 

Key  Codes  2  &  12  have  a  Handling  Variable  Set  first  digit  of  1.  This  designates 
that  the  Lessons  Learned  are  co-written  at  the  source  by  Lessons  Learned  System 
personnel.  This  represents  the  combination  of  an  Acquisition  characteristic  of  active  and 
a  possible  interpretation  of  the  Process  Relation  characteristic  of  embedded.  A  Lessons 
Learned  System  with  a  Handling  Variable  Set  of  Ix-xxx-xx  will  promote  the  receiving  of 
lessons  learned.  This  finding  also  supports  the  literature  review  statement  “information, 
you  have  to  get  it  yourself”  See  Table  II-2. 

Key  Codes  1,  8  &  10  are  at  a  mature  development  stage  and  have  high  upper 
management  support  along  with  a  combination  (including  embedded  for 
embedded/standalone  and  active  for  active/passive)  for  the  Process  Relation 
characteristic  and  the  Acquisition  characteristic. 

Key  Codes  1,  8,  9  &  13  are  mature  or  developing/mature  Lessons  Learned 
Systems.  These  four  Key  Codes  cited  the  importance  of  quality  lessons  learned  to  the 
value  or  usage  of  the  Lessons  Learned  System.  It  may  be  concluded  that  an  established 
history  of  quality  lessons  learned  will  increase  usage  including  the  submission  of  lessons 
learned. 
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The  Influence  Diagram  below  graphically  displays  influences  on  the  receiving  of 
lessons  learned  for  a  Lessons  Learned  System.  Refer  to  the  Chapter  III  for  the  meaning 
of  symbols. 

Figure  V-1  Receiving  Lessons  Learned  Influence  Diagram 


2.  Quality  of  Lessons  Learned 

No  existing  Lessons  Learned  System  identified  a  eoncern  about  the  quality  of  the 
Lessons  Learned  that  they  disseminate. 

Chapter  IV  identified  a  number  of  methods  used  to  review  lessons  learned  prior  to 
dissemination  to  insure  quality.  These  included: 

1.  Reviewed  at  source  by  supervisory  personnel  or  command  approved  or 
preliminary  level  of  review. 

2.  Reviewed  centrally  for  technical  adequacy  by  experts. 

3.  Verified  by  test,  or  feedback  once  disseminated. 

4.  Reviewed  centrally  for  editorial  adequacy.  This  included  root  cause 
analysis,  generic  filtering,  context  and  background  for  understanding, 
relevancy  and  public  relations. 

These  have  been  incorporated  into  the  Handling  Variable  Set  as  digits  2  thru  5. 
See  Table  V-l. 

The  choice  of  the  quality  values  (0  or  1  for  digits  2  thru  5)  for  the  Handling 
Variable  Set  for  Lessons  Learned  System  design  needs  to  be  fit  for  the  specific  Lessons 
Learned  System. 

The  first  quality  digit  (digit  2  of  the  Handling  Variable  Set),  reviewed  at  source 
by  supervisory  personnel  or  command  approved  or  preliminary  level  of  review,  is 
implemented  (value  of  1)  by  military  Lessons  Learned  Systems  (Key  Codes  1,  4,  5  &  6) 
and  Lessons  Learned  Systems  that  act  as  a  collecting  point  for  distributed  sources  (Key 
Codes  10,  11,  13  &  15).  The  benefits  of  implementing  the  choice  are  obtaining  lessons 
that  meet  a  certain  criteria  such  as  being  influenced  by  a  narrow  application  or  a  specific 
agenda  (Key  Code  1),  categorization  (Key  Code  13)  and  accuracy  (Key  Codes  6  &  15). 

The  second  quality  digit  (digit  3  of  the  Handling  Variable  Set),  reviewed  centrally 
for  technical  adequacy  by  experts,  is  implemented  (value  of  1)  by  Lessons  Learned 
Systems  (Key  Codes  4,  5,  8,  9,  10  &  13)  that  deal  with  a  subject  that  requires  a  special 
knowledge  to  properly  evaluate.  These  include  engineering  (Key  Codes  4  &  13),  medical 
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(Key  Code  5)  and  nuelear  (Key  Codes  8,  9  &  10).  Organizations  whose  subjects  are 
specialized  should  implement  a  review  by  experts  to  provide  assurance  of  quality. 

The  third  quality  digit  (digit  4  of  the  Handling  Variable  Set),  verified  by  test,  or 
feedback  once  disseminated,  is  implemented  (value  of  1)  by  Lessons  Learned  Systems 
(Key  Codes  4,  5,  &  13)  whose  subjects  are  such  that  confirmation  by  test  is  beneficial 
and  feasible.  The  implementation  of  engineering  (Key  Codes  4  &  13)  and  medical  (Key 
Code  5)  solutions  cannot  always  be  guaranteed  by  analysis  alone  and  often  empirical  data 
is  needed.  Verifying  by  test  is  not  always  feasible  with  some  engineering  fields  such  as 
nuclear,  where  minimizing  human  interaction  with  nuclear  materials  is  prudent.  Key 
Codes  8,  9,  10,  11  &  13  (in  part)  are  involved  with  the  nuclear  subject  and  do  not  verify 
accuracy  or  quality  by  test.  Key  Code  9  does  monitor  for  feedback  or  accuracy  once 
disseminated.  The  implementation  of  verified  by  test  (where  feasible),  or  feedback  once 
disseminated,  will  provide  assurance  of  accuracy  and  quality. 

The  final  quality  digit  (digit  5  of  the  Handling  Variable  Set),  reviewed  centrally 
for  editorial  adequacy  (root  cause,  generic,  background,  public  relations,  relevancy,  etc.), 
is  implemented  (value  of  1)  by  all  the  existing  Lessons  Learned  Systems  except  one  (Key 
Code  14).  Key  Code  14  is  a  new  Lessons  Learned  System  and  is  attempting  to  develop 
quality  by  open  (through  web  site  postings)  discussion.  To  insure  some  level  of  accuracy 
and  quality,  the  Lessons  Learned  System  should  implement  an  editorial  review.  This 
finding  supports  a  number  of  literature  statements.  See  Table  11-2. 

The  above  provides  guidance  on  choosing  the  Handling  Variable  Set  to  meet 
quality  needs.  To  characterize  the  level  of  quality  implemented  by  Lessons  Learned 
Systems  through  the  Handling  characteristic  in  a  more  general  sense,  the  quality  digits  of 
the  Handling  Variable  Set  can  be  added  to  give  a  general  level.  For  example.  Key  Code 
5  with  a  Handling  Variable  Set  of  0-1 1 1 1-01  can  be  condensed  to  0-4-1  or  041  indicating 
a  high  degree  of  effort  concerning  quality.  Key  Code  3  with  a  Handling  Variable  Set  of 
0-0001-00  can  be  condensed  to  0-1-0  or  010  indicating  a  lesser  effort  concerning  quality. 

As  a  general  observation.  Chapter  IV  provided  three  criteria  that  could  influence 
the  choice  of  quality  values  for  the  Handling  Variable  Set.  These  included  the  goals  of 
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Lessons  Learned  System,  resourees  available  to  the  Lessons  Learned  System  and 
responsibility  level  of  the  Lessons  Learned  System. 


Table  V-3  below  lists  qualitative  values  for  the  three  criteria.  The  information 
was  abstracted  from  Tables  IV-3  and  Table  IV-7. 


Table  V-3  Quality  Influences  of  Existing  Lessons  Learned  Systems 

Key 

Code 

Handling 
Variable  Set 

Quality 

Sum 

Goal 

Resources 

Responsibility 

1 

0-1001-01 

2 

operational 

medium 

medium 

2 

1-0101-10 

2 

operational 

medium 

low 

3 

0-0001-00 

1 

safety 

low 

low 

4 

0-1111-01 

4 

engineering 

medium 

high 

5 

0-1111-01 

4 

medical 

medium 

high 

6 

0-1001-00 

2 

operational 

low 

medium 

7 

0-0001-00 

1 

operational 

medium 

low 

8 

0-1101-00 

3 

nuclear  safety 

medium 

medium 

9 

0-0101-00 

2 

nuclear  safety 

medium 

medium 

10 

0-1101-00 

3 

nuclear  safety 

medium 

medium 

11 

0-1001-00 

2 

nuclear  safety 

medium 

low 

12 

1-1001-10147 

2 

operational 

medium 

low 

13 

0-1111-11 

4 

engineering 

medium 

medium 

14 

0-0001-00 

1 

operational 

low 

low 

15 

0-1001-10 

2 

engineering 

medium 

medium 

147  In  analyzing  this  Lessons  Learned  System,  the  hired  consultants  are  considered  as  Lessons 
Learned  System  agents  and  their  actions  are  the  actions  of  the  Lessons  Learned  System 
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Table  V-3  indicates  that  Lessons  Learned  Systems  whose  goal  is  medical  or 
engineering,  have  high  level  of  responsibility  or  involvement  in  policy  implementation 
and  have  available  resources  have  the  highest  general  level  of  quality  review.  It  also 
shows  that  the  absence  of  one  criterion  (subject,  resources  &  responsibility)  results  in 
reduced  levels  of  handling  in  terms  of  quality  of  lessons.  Key  Code  13  has  made  the 
quality  of  lessons  a  self  imposed  requirement. 

This  can  be  graphically  displayed  in  an  Influence  Diagram.  Refer  to  the  Chapter 
III  for  the  meaning  of  symbols. 


Figure  V-2  Influences  on  the  Level  of  Quality  Review 
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The  design  of  the  Handling  eharacteristic  of  a  Lessons  Learned  System  for  quality 
is  first  based  on  the  level  required  per  the  criteria  of  Figure  2.  Once  the  level  is 
established,  the  methods  of  handling  associated  with  quality  of  the  Handling  Variable  Set 
can  be  chosen  where  feasible. 

The  particulars  of  the  editorial  review  must  be  fitted  to  the  uniqueness  of  the 
organization. 

3.  Implementation  of  Lessons  Learned 

There  are  two  methods  of  handling  identified  in  Table  IV-2  that  support 
implementation  of  Lessons  Learned.  These  are: 

1 .  Reviewed  centrally  or  otherwise  by  potential  users/policy  implementers 
prior  to  dissemination. 

2.  Identifies  target  for  dissemination,  may  also  require  a  response. 

These  have  been  incorporated  into  Table  V-1  variable  numbers  six  and  seven. 

Key  Code  12  identified  that  it  had  no  present  concerns,  that  it  was  satisfied  with 
the  implementation  of  its  Lessons  Learned,  see  Table  IV-3.  A  review  of  its  Handling 
Variable  Set,  see  Table  V-1,  indicates  a  1  for  the  sixth  digit  signifying  that  lessons  are 
reviewed  centrally  or  otherwise  by  potential  users/policy  implementers  prior  to 
dissemination.  Key  Codes  13  &  15  also  implements  this  handling  method.  Key  Code  13, 
in  Chapter  IV  states  that  there  is  a  credibility  that  exists  when  the  management  board 
endorses  a  lesson  and  this  encourages  ...  use. 1 48  Key  Code  15,  in  Chapter  IV,  states, 
there  is  evidence  that  the  database  is  being  addressed  by  the  ...  for  lessons  learned.  149 
Key  Codes  13  &  15  do  not  have  a  concern  about  the  implementation  of  lessons  learned. 

The  remaining  Key  Code  where  lessons  are  reviewed  centrally  or  otherwise  by 
potential  users/policy  implementers  prior  to  dissemination  is  Key  Code  2.  Key  Code  2 
has  a  present  concern  of  implementing  lessons.  It  is  identified  in  Chapter  IV  that  one 
purpose  of  their  Lessons  Learned  System  is  to  provide  the  lessons  learned  to  policy 


148  see  page  75 

149  see  page  79 


105 


makers  who  can  implement  the  lessons  learned.  This  is  done  by  policy  makers 
participating  in  the  validation  process.  150 

There  is  evidence  that  a  Lessons  Learned  System  employing  a  Handling  Variable 
Set  where  the  sixth  digit  is  one,  reviewed  centrally  or  otherwise  by  potential  users/policy 
implementers  prior  to  dissemination,  will  have  a  positive  effect  on  an  organization 
implementing  their  Lessons  Learned.  This  finding  supports  the  literature  statements 
about  “management  support  and  involvement”  and  “users  participate  in  process/design.” 
See  Table  II-2. 

Key  Codes  3,  6,  8,  9,  10,  1 1  &  14  have  zeros  for  the  sixth  and  seventh  digit  of  the 
Variable  Handling  Set,  see  Table  V-2.  A  one  for  digit  seven  represents  the  Lessons 
Learned  System  identifying  a  target  for  dissemination,  may  also  require  response.  Key 
Codes  8,  9  &  10  identified  their  present  concern  as  implementing  lessons,  see  Table  IV-3. 
Key  Codes  3,  6,  11  &  14  did  not  identify  their  present  concern  as  implementing  lessons 
but  rather  receiving  lessons.  Key  Codes  3,  6  &  14  have  an  Acquisition  characteristic  of 
passive,  see  Table  IV-4.  Key  Codes  3,  6,  11  &  14  also  have  a  0-xxxx-xx  Handling 
Variable  Set,  see  Table  V-2,  so  having  a  present  concern  of  receiving  lessons  and  not 
implementing  lessons  may  be  a  case  of  one  before  the  other. 

Key  Code  1  is  a  Lessons  Learned  System  that  has  a  one  in  the  seventh  digit  and 
also  identified  implementing  lessons  as  its  present  concern,  see  Table  V-2  and  Table  IV- 
3.  It  is  noted  that  Key  Code  I  could  be  classified  as  having  a  rigid  qualitative  value  for 
the  Organizational  Type  characteristic,  see  Chapter  II,  Section  E.  This  implies  that  the 
concern  of  implementing  lessons  may  be  more  dependent  on  the  Organizational  Type 
characteristic  than  on  handling  aspects. 

Key  Code  13  is  a  Lessons  Learned  System  that  has  a  one  in  the  seventh  digit,  see 
Table  V-2.  Key  Code  13  identified  in  Chapter  IV  that  their  identification  of  a  target  for 
dissemination,  a  one  for  digit  seven  of  the  Handling  Variable  Set,  includes  pro-active 
involvement.  The  target  is  required  to  respond  and  the  movement  of  the  Lessons  Learned 
is  tracked  to  insure  required  actions  are  taken.  This  finding  supports  the  literature 
statement  about  “management  support  and  involvement.”  See  Table  II-2. 


150  see  page  56 
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The  influence  diagram  below  graphically  displays  influences  on  the 
implementation  of  lessons  learned  for  a  Lessons  Learned  System.  Refer  to  the  Chapter 
III  for  the  meaning  of  symbols. 


Figure  V-3  Implementing  Lessons  Learned  Influence  Diagram 


4,  Summary 

Sections  1  Receiving  Lessons,  Section  2  Quality  of  Lessons  Learned  and  Section 
3  Implementation  of  Lessons  Learned  have  provided  guidance  for  Lessons  Learned 
Systems  designers  in  the  areas  of  receiving  lessons,  insuring  that  lessons  learned  are  of 
high  quality  and  the  usage  or  implementation  of  lessons  learned  with  an  emphasis  on  the 
Handling  characteristic. 
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E.  CONCLUSION 

The  primary  research  question  of  this  thesis  is:  How  may  a  Lessons  Learned 
System  be  characterized  according  to  the  handling  of  lessons  processing,  and  how  may 
such  a  characterization  be  applied  to  Lessons  Learned  System  design  or  architecture? 

Section  C  provided  a  coding  of  existing  handling  methods.  The  coding  is  a 
seven-digit  number,  expressed  x-xxxx-xx,  where  a  one  represents  an  action  and  a  zero 
represents  omission.  The  separation  by  dashes  allows  the  handling  to  be  decomposed 
into  actions  that  affect  receiving  lessons,  quality  of  lessons  and  handling  lessons 
concerning  implementation.  The  coding  can  further  be  condensed  by  adding  the  values 
of  the  separated  section  to  form  a  three-digit  number.  For  example  a  Handling  Variable 
Set  of  1-1010-10  can  be  condensed  to  1-2-1  or  121.  This  would  provide  a  quick  measure 
of  the  effort  of  quality.  This  coding,  the  Handling  Variable  Set,  provides  one  answer  to: 
How  may  a  Lessons  Learned  System  be  characterized  according  to  the  handling  of 
lessons  processing?,  the  first  part  of  the  primary  research  question. 

Section  D  provided  qualitative  analysis  based  on  existing  Lessons  Learned 
Systems  to  answer  the  second  part  of  the  primary  research  question:  how  may  such  a 
characterization  be  applied  to  Lessons  Learned  System  design  or  architecture?  The 
analysis  provided  the  effects  of  the  Handling  Variable  Set  on  receiving  lessons,  quality  of 
lessons  learned  and  implementation  of  lessons  learned. 

For  receiving  lessons.  Figure  V-1  provided  a  cause  and  effect  relationship 
between  the  Handling  Variable  Set  and  receiving  lessons. 

For  the  quality  of  lessons.  Figure  V-2  provided  an  estimate  of  what  an  appropriate 
level  of  quality  should  or  could  be.  The  Handling  Variable  Set,  particularly  the  quality 
section,  can  then  be  examined  against  this  level. 
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For  implementation  of  lessons  learned,  Figure  V-3  provided  a  eause  and  effeet 
relationship  between  the  Handling  Variable  Set  and  the  implementation  of  Lessons 
Learned. 

The  eombined  use  of  the  Handling  Variable  Set  and  Figures  V-1,  V-2  and  V-3 
can  be  used  toward  Lessons  Learned  System  design  or  architecture. 
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VI.  APPLICATION  TO  SUPSHIP  GROTON 


A.  INTRODUCTION 

This  chapter  applies  the  eharaeterization  of  lessons  handling  and  its  applieation  to 
design  to  the  SUPSHIP  Groton  Lessons  Learned  System.  The  SUPSHIP  Groton  Lessons 
Learned  System  is  a  new  developing  system.  The  baekground  for  the  SUPSHIP  Groton 
Lessons  Learned  System  is  provided  in  Seetion  B. 

A  eharaeterization  of  lessons  handling  is  the  Handling  Variable  Set  as  was 
developed  in  Chapter  V.  The  Handling  Variable  Set  and  the  Influenee  Diagrams  of 
Chapter  V  ean  be  used  to  evaluate  the  design  or  arehiteeture  of  a  Lessons  Learned 
System  with  respect  to  its  basic  tasks.  The  basie  tasks  are  reeeiving  lessons,  developing  a 
quality  lesson  for  dissemination,  and  dissemination  with  the  goal  of  implementation. 

Seetion  C  provides  the  analysis  and  Seetion  D  provides  reeommendations  based 
on  the  analysis  and  additional  reeommendations  based  on  Chapters  II  and  IV. 


B,  SUPSHIP  GROTON  LESSONS  LEARNED  SYSTEM 

SUPSHIP  Groton  is  a  Naval  Sea  Systems  Command  (NAVSEA)  field 
organization  loeated  in  Groton,  CT.  SUPSHIP  Groton  represents  NAVSEA  and  oversees 
nuelear  submarine  design,  new  eonstruction  and  submarine  repair  efforts  of  the  Eleetrie 
Boat  Corporation.  151 

Past  praetice  had  been  for  NAVSEA  to  send  a  team  to  SUPSHIP  Groton  to  audit 
SUPSHIP  Groton  operations.  The  audit  eould  eoneentrate  on  any  area.  A  few  years  ago, 
NAVSEA  began  an  initiative  to  align  its  operations  at  headquarters  and  at  field  offiees 
more  elosely  with  the  best  business  praetiees  of  the  private  seetor.  As  a  eonsequence,  the 
audits  were  supplemented  with  a  NAVSEA  evaluation  of  SUPSHIP  Groton  to  the  eriteria 
of  the  Baldrige  National  Quality  Program.  Eor  SUPSHIP  Groton,  this  included  a  Unit 
Self  Assessment  and  a  Command  Performanee  Inspeetion. 

151  SUPSHIP  Groton  Instruction  5224.1  of  28  Feb  02 
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As  a  result  of  a  self-analysis,  it  was  coneluded  that  SUPSHIP  Groton  eould 
improve  in  the  area  of  Customer  and  Market  Focus,  particularly  in  Customer  Relations. 
As  a  result,  a  Customer  Focus  Policy  was  issued  to  address  this  improvement.  Part  of  the 
improvement  policy  was  the  development  of  an  evaluation  process  entitled  AFTER 
(After  the  Fire  Take  Time  to  Evaluate  and  Review). 

AETER  is  a  structured  post  crisis  or  post  key  event  evaluation  with  the  goal  of 
capturing  what  SUPSHIP  Groton  did  right  and  where  efforts  need  improvement.  The 
goal  of  AETER  is  to  ensure  what  SUPSHIP  Groton  did  well  during  the  “fire”  is 
embedded  in  our  normal  work  processes  and  to  modify  any  aspect  of  SUPSHIP  Groton 
operation  or  process  to  improve  future  performance.! 52 

This  fits  the  general  definition  of  a  lessons  learned  system.  The  judgement  of 
doing  well  or  not  doing  well  is  based  on  customer  satisfaction.  A  complaint  or 
dissatisfaction  by  the  customer  would  be  a  negative  experience.  Eikewise,  a  satisfied 
customer  would  be  a  success.  AETER  is  new  and  is  still  under  development  with  many 
details  of  operation  needed  to  be  determined.  There  are  no  procedures  for  operation. 
There  is  only  a  temporary  repository  for  “lessons”,  that  being  Word  software.  The  details 
of  dissemination  and  implementation  have  not  been  worked  out. 

AETER  has  not  yet  been  used.  The  Customer  Eocus  Policy  includes  other 
programs  to  promote  customer  relations.  These  have  begun  to  operate  and  customer 
input  has  been  collected  and  although  SUPSHIP  Groton  personal  have  addressed  the 
specific  customer  complaints,  no  root  cause  or  general  lesson  learned  that  could  be  used 
to  determine  correct  processes  have  been  determined  from  the  complaint. 

SUPSHIP  Groton  customers  include  NAVSEA  Washington  DC  Corporate 
Headquarters,  Program  Executive  Offices,  Commander  Submarine  Eorce,  U.S.  Atlantic 
Elect,  the  Officers  and  Crews  of  pre-commissioned  submarines  under  construction  at 
Electric  Boat,  the  Officers  and  Crews  of  commissioned  submarines  in  overhaul  or  repair 
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at  Electric  Boat  and  at  the  Naval  Submarine  Base  New  London,  Groton,  CT  under 
Electric  Boat  contract  and  Naval  Laboratories.! 53 

ALTER  is  not  a  continuous  operating  system.  The  system  is  initiated  by 
management  calling  for  ALTER  action.  It  is  planned  that  this  calling  for  action  will  be 
after  an  unusual  event  that  required  unplanned  SUPSHIP  Groton  action  or  a  key  event 
such  as  the  completion  of  an  overhaul,  a  design  review,  etc.  The  basic  process  is  first  a 
meeting  to  discuss  lessons  learned  from  the  action  or  involvement  in  a  key  event.  This 
meeting  is  to  include  solicited  dispositions  and  interviews  from  the  customers  that  were 
involved.  A  report  is  written  containing  the  findings  and  posted  on  the  SUPSHIP  Groton 
Intranet.  The  report  will  include  specific  action  items  or  recommendations  for  study. 

As  stated  earlier,  the  ALTER  system  has  yet  to  be  implemented.  It  is  expected 
that  the  process  will  be  adjusted  as  experience  with  the  ALTER  system  is  gained. 


C.  ANALYSIS 

The  first  task  of  the  analysis  is  to  determine  the  Handling  Variable  Set  for  the 
SUPSHIP  Groton  Lessons  Learned  System.  Lrom  the  description  in  Section  B,  the 
Handling  Variable  Set  is  0-0001-10.  This  corresponds  to  a  Lessons  Learned  System 
where  lessons  are  not  co-written  at  source  by  Lessons  Learned  System  personnel,  there  is 
no  preliminary  supervisory  approval,  no  review  by  experts  as  to  accuracy,  no  verification 
by  test  and  no  target  for  dissemination  requiring  a  response.  There  is  a  central  review  for 
accuracy  by  potential  policy  makers  prior  to  issuing  a  report. 

The  second  task  is  to  use  the  Handling  Variable  Set  and  the  Influence  Diagrams 
of  Chapter  V  to  predict  the  performance  of  the  SUPSHIP  Groton  Lessons  Learned 
System  with  respect  to  the  three  basic  tasks  of  a  Lessons  Learned  System.  The  three 
tasks  are  receiving  lessons,  developing  quality  lessons  prior  to  dissemination  and 
disseminating  lessons  with  the  goal  of  implementation. 


153  SUPSHIP  Groton  Instruction  5224.1  of  28  Feb  02 


113 


1.  Receiving  Lessons 

From  Figure  V-1,  due  to  an  absence  of  a  one  in  the  first  digit  of  the  Handling 
Variable  Set,  the  empirical  evidence  suggests  that  the  SUPSHIP  Groton  Lessons  Learned 
System  will  experience  a  concern  about  receiving  lessons.  Reliance  on  the  customer 
retaining  the  lessons  until  the  meeting  or  the  local  SUPSHIP  supervision  collecting 
lessons  as  a  collateral  duty  will  not  be  sufficient.  This  was  suggested  in  Chapter  II,  see 
the  Table  II-2,  and  supported  empirically  in  Chapter  V,  Section  D.l. 

The  characteristic  Acquisition  describes  how  lessons  are  obtained.  A  Lessons 
Learned  System  with  an  Acquisition  characteristic  qualitative  value  of  active  seeks  out 
lessons  by  incorporating  itself  into  the  organization’s  operations  or  other  active  searches. 
Although  the  SUPSHIP  Groton  Lessons  Learned  System  does  actively  search  for  lessons 
by  interviewing  the  customer,  because  the  interview  may  be  some  time  after  the  events, 
the  Acquisition  qualitative  value  of  active  cannot  be  considered  a  strong  value.  Without  a 
strong  or  certain  value  of  active.  Figure  V-1  implies  that  receiving  lessons  may  be  a 
concern. 

The  characteristic  of  Process  Relation  is  a  measure  of  how  integrated  the  Lessons 
Learned  System  is  with  an  organization’s  operation.  A  qualitative  value  of  embedded 
requires  as  part  of  operations  the  recording  of  lessons  learned.  The  SUPSHIP  Groton 
Lessons  Learned  System  is  not  embedded  but  is  standalone.  Figure  V-1  implies  that 
receiving  lessons  may  be  a  concern. 

The  other  influences  affecting  receiving  lessons  from  Figure  V-1  are  Lower 
management  enforcement  on  completing  lessons  learned  paperwork.  Upper  management 
communication  of  the  importance  of  lessons  learned  and  Established  history  of  quality  of 
lessons  learned.  These  are  not  decision  variables  available  to  the  Lessons  Learned 
System  and  are  outside  or  environment  influences,  see  Table  II- 1.  The  first  influence  is 
not  applicable  as  there  is  no  lesson  learned  paper  work,  as  there  would  be  with  a  Lessons 
Learned  System  with  an  embedded  qualitative  value  for  its  Process  Relation 
characteristic.  The  second  influence  has  not  occurred  as  yet  and  the  third  influence  is  not 
applicable  due  to  the  newness  of  the  SUPSHIP  Groton  Lessons  Learned  System. 
Therefore,  these  cannot  have  a  positive  influence  on  receiving  lessons. 
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Based  on  the  Figure  V-1  Influence  Diagram  and  the  Handling  Variable  Set,  the 
analysis  suggests  that  receiving  lessons  will  be  a  concern. 

2.  Quality  of  Lessons  Learned 

The  level  of  quality  required  or  possible  by  a  Lessons  Learned  System  is  first 
determined  by  the  use  of  Figure  V-2.  A  low  value  of  any  of  the  three  influences  will 
predict  a  lack  of  high  quality.  The  two  environmental  influences  from  Figure  V-2  are 
Resources  available  to  the  Lessons  Learned  System  and  Level  of  responsibility  or  power 
of  the  Lessons  Learned  System  concerning  policy  implementation.  Since  these  are 
probably  moderate  at  best,  the  quality  may  not  be  high.  With  respect  to  the  two 
influences  above,  a  moderate  level  would  be  employees  participating  in  the  Lessons 
Learned  System  as  collateral  duty  with  little  guarantee  that  their  products  will  be  used. 

The  third  influence  is  the  goals  of  the  Lesson  Learned  System,  a  decision 
variable.  The  interpretation  of  goals  in  Figure  V-2  was  based  on  the  subject  of  the 
lessons.  Lessons  Learned  Systems  dealing  with  medical,  engineering  and  nuclear  had  a 
high  level  of  quality  development.  This  was  required  based  on  the  consequences  of 
inaccuracy.  The  SUPSHIP  Groton  Lessons  Learned  System  has  not  clearly  defined  the 
subject  matter  that  it  will  be  concentrating  on,  only  customer  satisfaction.  Since  the 
subject  matter  may  likely  be  operational,  a  high  level  of  quality  development  may  not  be 
necessary. 

Based  on  Figure  V-2,  the  quality  of  lessons  required  may  not  be  great,  from  a 
relative  point  of  view. 

The  Handling  Variable  Set,  0-0001-10,  with  a  quality  section  of  0001,  is  probably 
acceptable.  This  corresponds  to  an  editorial  type  review.  The  omission  of  a  prior 
command  approval,  verification  by  technical  experts  and  verification  by  test  is  probably 
acceptable. 

The  use  of  experts  to  provide  accuracy  in  technical  areas  is  an  available  resource 
for  the  SUPSHIP  Groton  Lessons  Learned  System  and  would  probably  be  implemented  if 
necessary.  Since  the  goal  of  the  SUPSHIP  Groton  Lessons  Learned  System  is  customer 
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satisfaction  and  the  customer  is  a  member  of  the  review  team,  quality  of  Lessons  Learned 
in  terms  of  accuracy  should  not  be  a  concern. 

An  editorial  review  alone  may  provide  quality  lessons  but  is  dependent  on  the 
editorial  review  being  of  high  quality  itself.  The  Handling  Variable  Set  and  Figure  V-2 
do  not  provide  guidance  or  evaluation  on  what  is  a  high  quality  editorial  review  because  a 
high  quality  review  is  so  dependent  on  being  fitted  to  the  uniqueness  of  the  Lessons 
Learned  System  subject  matter,  its  goals  and  its  organization.  This  is  further  discussed  in 
Section  E. 


3,  Implementation  of  Lessons  Learned 

From  Figure  V-3,  and  a  one  in  the  sixth  digit  of  the  Handling  Variable  Set 
signifying  reviewed  by  poliey  implementers,  the  implementation  of  lessons  learned 
should  not  be  a  problem.  A  one  in  the  sixth  digit  of  the  Handling  Variable  Set  is  a  strong 
indicator,  both  theoretically  and  empirically,  that  Fessons  Feamed  will  be  implemented. 

A  one  in  the  seventh  digit  indieates  that  a  Fessons  Fearned  System  identify  a 
target  for  dissemination  and  may  also  require  response.  The  Handling  Variable  Set  for 
the  SUPSHIP  Groton  Fessons  Learned  System  is  a  zero  for  the  seventh  digit.  This 
omission  will  not  have  a  positive  effect  on  the  implementation  of  lessons. 

The  lone  environmental  factor  effect  the  implementation  of  lessons,  from  Figure 
V-3  is  the  Organizational  Type  eharaeteristie.  SUPSHIP  Groton  is  judged  not  to  be  rigid, 
therefore  the  Handling  Variable  Set  influenees  on  the  implementation  of  lessons  will  not 
be  countered. 

Based  on  the  Handling  Variable  Set  and  Figure  V-3,  the  implementation  of 
lessons  is  probably  not  a  concern.  The  probability  of  lessons  being  implemented  eould 
be  higher  if  the  SUPSHIP  Groton  Fessons  Fearned  System  incorporated  the  praetiee  of 
identifying  targets  for  dissemination  and  requiring  a  response. 
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D,  RECOMMENDATIONS 


In  fairness  to  SUPSHIP  Groton,  the  intent  of  AFTER  may  not  to  be  a  Lessons 
Learned  System  but  only  one  proeess  in  a  total  plan  to  inerease  SUPSHIP  Groton 
efficieney  and  eustomer  satisfaetion.  The  following  recommendations,  though,  are  based 
on  the  intent  of  ALTER  being  a  Lessons  Learned  System. 

Without  the  benefit  of  operating  experience  and  for  a  first  effort  design,  the 
SUPSHIP  Groton  Lessons  Learned  System  is  a  fairly  good  design.  In  its  simplicity,  it 
has  incorporated  methods  (particularly  lessons  being  reviewed  by  policy  implementers 
prior  to  dissemination)  that  empirical  evidence  suggests  is  beneficial  to  a  Lessons 
Learned  System  Success.  However  it  is  likely  that  not  all  appropriate  lessons  will  enter 
the  system. 

1.  Recommendations  Based  on  Analysis 

The  recommendation  is  that  SUPSHIP  Groton  establish  a  position  whose 
responsibility  it  is  to  collect  lessons  learned  as  a  sole  activity  on  a  daily  basis  from  the 
different  activities  that  are  occurring.  Along  with  this,  it  is  necessary  that  management 
communicate  that  minor  infringements  to  occurring  work  activities  are  necessary  for 
long-term  growth. 

Actively  collecting  lessons  in  this  manner  will  assure  lessons  are  received  into  the 
system  and  coupled  with  the  existing  SUPSHIP  Groton  Lessons  Learned  System 
architecture,  the  SUPSHIP  Groton  Lessons  Learned  System  should  be  successful. 

Expending  resources  for  this  recommendation  may  be  a  concern.  In  support  of 
the  expenditure  is  the  argument  that  modern  businesses  view  Knowledge  Management 
(of  which  a  Lessons  Learned  System  is  a  part)  and  Organizational  Learning  as  essential 
to  increased  efficiency  and  competitiveness  in  the  business  world.  If  resources  are 
limited,  it  is  suggested  that  the  collection  of  lessons  as  a  collateral  duty  by  local 
SUPSHIP  supervision  for  an  activity  or  an  event  be  proactively  enforced  by  management. 
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There  are  no  recommendations  based  on  the  analysis  concerning  quality  of 
lessons.  The  use  of  editorial  review  is  considered  sufficient  to  provide  adequate  quality 
for  the  intended  purpose  of  the  lessons.  Again,  the  use  of  experts  to  provide  accuracy  in 
technical  areas  is  an  available  resource  for  the  SUPSHIP  Groton  Lessons  Learned  System 
and  would  probably  be  implemented  if  necessary. 

The  implementation  of  lessons  should  not  be  a  major  concern.  The  use  of  policy 
implementers  to  review  the  lessons  and  write  the  report  is  a  positive  influence  on  the 
implementation  of  lessons.  The  probability  that  lessons  would  be  implemented  can  be 
increased  though  by  targeting  places  of  dissemination  and  requiring  a  response  and 
should  be  a  consideration. 

2.  Other  Recommendations 

The  Handling  Variable  Set  and  the  use  of  the  Influence  Diagrams  for  analysis  and 
design  or  architecture  is  predominantly  focused  on  the  Handling  characteristic.  There  are 
other  aspects  of  design,  beyond  the  Handling  characteristic,  that  should  be  considered. 
Further,  the  details  of  the  editorial  review  for  quality,  are  also  beyond  the  scope  of  the 
Handling  Variable  Set  and  the  use  of  the  Influence  Diagrams.  Fortunately,  the 
information  in  Chapters  II  and  IV  can  be  used  to  provide  guidance  and  other 
rec  ommendations . 

The  editorial  process  used  to  provide  quality  of  lessons  needs  to  be  specifically  fit 
to  the  organization  and  the  subject.  The  first  task  is  to  focus  on  the  subject.  The  present 
plan  is  to  query  the  customer  on  what  SUPSHIP  Groton  did  well  and  did  not  do  well  in 
performing  some  service  to  the  customer.  Once  these  are  received,  they  must  be  filtered 
to  determine  if  there  is  a  root  cause  that  can  be  addressed  or  if  it  is  an  isolated  incident.  If 
there  is  a  root  cause  then  there  can  be  a  change  that  will  prevent  the  same  problem  in  the 
future. 

A  solution  to  the  root  cause  is  not  an  easy  task  and  may  be  more  of  an  art  than  a 
science.  An  assumption  that  must  be  made  is  that  someone  in  the  organization  knows  the 
solution  and  it’s  a  matter  of  finding  that  person.  For  example,  a  customer  may  state  that 
the  turnaround  time  on  technical  documents  is  too  long.  From  a  manager’s  perspective, 
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the  root  cause  may  be  too  long  a  time  to  transfer  the  technical  documents  from  point  to 
point  along  the  approval  path  and  suggest  quicker  electronic  transfer  methods  as  the 
solution.  A  person  working  the  technical  document  may  see  the  root  cause  as  lack  of 
information  transfer.  For  example,  a  customer  may  perform  a  determination  as  to  who  is 
the  approval  authority  for  a  technical  document  but  only  include  the  conclusion  in  the 
technical  document  leaving  the  receiver  of  the  technical  document  to  repeat  the  exercise 
to  satisfy  his  management  of  the  conclusion.  Or  the  customer  may  after  painstakingly 
navigating  through  many  narrow  computer  screen  views  of  an  electronic  drawing  to  find 
a  detail,  only  cite  the  drawing  and  force  the  receiver  of  the  technical  document  to  repeat 
the  tedious  work  that  has  already  been  done  but  not  transferred.  Once  the  person  or 
collection  of  people  who  know  the  root  cause  or  a  potential  solution  to  a  customer 
concern  is  identified,  then  the  solution  must  be  transformed  from  tacit  to  explicit 
knowledge  so  that  it  can  be  transferred  to  the  organization. 

Explicit  knowledge  is  written  knowledge  or  perhaps  video  knowledge. 
Suggestions  from  Chapters  II  and  IV  to  make  knowledge  more  explicit  is  to  provide 
context  and  remove  irrelevancy.  This  is  subjective  though,  as  the  goal  of  explicit 
knowledge  is  to  develop  tacit  knowledge  in  the  receiver  and  the  correct  explicit  form  ti 
initiate  this  is  probably  different  for  different  receivers.  The  SUPSHIP  Groton  Lessons 
Learned  System  does  not  presently  identify  or  target  a  receiver. 

The  above  involves  the  quality  of  lessons. 

The  present  plan  for  the  SUPSHIP  Groton  Lessons  Learned  System  is  to  collect 
Lessons  Learned  in  a  report.  Although  the  report  is  planned  to  be  written  by  policy 
implementers  which  empirical  evidence  indicates  promotes  implementation,  there  may  be 
some  policy  implementers  that  are  not  involved  and  desired  implementation  by  these  may 
not  occur. 

Adopting  the  International  Space  Station  method  of  targeting  the  lesson  learned  to 
a  policy  maker  and  requiring  a  response  is  a  method  that  further  promotes 
implementation.  This  should  be  done  with  a  software  that  is  comfortable  to  the  receiver 
as  suggested  in  Chapter  IV. 


II9 


One  suggestion  from  Chapter  II  is  that  a  Lessons  Learned  System  be  embedded  in 
the  operations  of  the  organization.! 54  A  suggestion  from  Chapter  IV  is  to  make  the 
Lessons  Learned  System  fit  with  the  tools  your  workers  use  every  day.! 55  An  implication 
of  these  suggestion  is  that  the  Lessons  Learned  Systems  be  continuously  operating.  It  is 
recommended  that  the  SUPSHIP  Groton  Lessons  Learned  System  be  a  continuously 
operating  system  and  not  just  called  into  operation  when  desired. 

Chapters  II  and  IV  also  suggested  the  importance  of  information  technology.! 56 
The  SUPSHIP  Groton  Lessons  Learned  System  currently  uses  Word  software  to  retain 
lessons.  This  software  has  advantages  in  terms  of  familiarity  and  its  inherent  word 
processing.  However  it  lacks  in  its  ability  to  search  or  query.  It  is  recommended  that  the 
Lessons  Learned  be  retained  in  Word  but  that  they  be  serialized  and  Access  software  be 
used  to  create  a  database  of  these  Lessons  Learned  that  can  be  queried  as  to  subject,  etc. 
so  they  may  be  found  and  applied  as  reference  for  future  work  endeavors. 


E,  CONCLUSIONS 

This  chapter  demonstrated  the  characterization  according  to  the  handling  of 
lessons  processing  of  a  Lessons  Learned  System  by  a  Handling  Variable  Set  and  its 
application  to  Lessons  Learned  System  design  or  architecture  by  use  of  the  Influence 
Diagrams  for  a  new  developing  Lessons  Learned  System.  The  proposed  design  is 
evaluated  against  the  tasks  of  a  Lessons  Learned  System.  Those  tasks  are  receiving 
lessons,  developing  quality  of  the  lessons  prior  to  dissemination  and  dissemination  with 
the  goal  of  implementation.  Also  included  was  a  broader  evaluation  based  on  Chapters  II 
and  IV. 


154  see  page  18 

155  see  page  67 

156  see  pages  16  and  72 
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The  result  of  the  analysis  is  that  AFTER,  although  not  intended  to  be  a  complete 
Lessons  Learned  System,  has  a  fairly  good  design,  except  possibly  there  will  be  a 
concern  about  receiving  lessons.  The  recommendations  may  help  the  design,  but 
pragmatically  should  only  be  considered  after  operation  experience  suggests 
improvement  is  needed,  particularly  concerning  the  recommendation  about  receiving 
lessons,  as  the  recommendations  would  require  resources. 
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VII.  CONCLUSION 


A.  THE  PRIMARY  RESEARCH  QUESTION 

The  primary  research  question  is:  How  may  a  Lessons  Learned  System  be 
characterized  according  to  the  handling  of  lessons  processing,  and  how  may  such  a 
characterization  be  applied  to  Lessons  Learned  System  design  or  architecture? 

One  method  of  characterizing  the  handling  of  lessons  processing  is  by  the 
Handling  Variable  Set.  This  coding  encompasses  the  combinations  of  activities 
associated  with  the  handling  of  lessons  processing.  It  is  not  qualitative  in  nature  by  itself 
but  allows  a  condensation  of  what  could  be  a  large  qualitative  description.  This  coding 
separates  actions  that  effect  the  three  tasks  of  a  Lessons  Learned  System. 

It  was  found  that  some  handling  actions  effect  receiving  lessons,  some  effect  the 
quality  of  the  lessons  and  some  effect  implementation  after  dissemination.  Two  of  the 
Influence  Diagrams  of  Chapter  V  provide  the  cause  and  effect  relationship  that  exists 
between  the  Handling  Variable  Set  and  the  tasks  of  receiving  lessons  and  implementation 
after  dissemination.  The  third  Influence  Diagram  of  Chapter  V  provides  a  guide  to  the 
appropriate  level  of  quality. 

The  level  of  quality  and  the  methods  of  obtaining  quality  are  dependent  on  the 
organization  and  the  subject  of  its  lessons.  There  are  some  general  rules  for  quality  and 
these  have  been  expressed  in  Chapter  VI. 

The  Handling  Variable  Set  and  the  Influence  Diagrams  of  Chapter  V  allow  the 
characterization  to  be  used  in  Lessons  Learned  System  Design  or  architecture,  the  second 
part  of  the  primary  research  question.  This  was  demonstrated  in  Chapter  VI. 
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B.  LIMITATIONS  OF  THE  RESEARCH 

The  research  was  based  on  a  sample  of  Lessons  Learned  Systems.  It  is  unknown 
to  what  degree  this  sample  represents  the  whole  of  Lessons  Learned  Systems.  Further  the 
data  was  based  on  the  Lessons  Learned  System’s  point  of  contact  personnel  experiences 
or  opinions  and  did  not  necessarily  represent  the  respective  organization’s  official 
position. 

The  abstraction  of  data  was  subjective  as  it  was  of  a  qualitative  nature.  Lessons 
Learned  Systems  point  of  contact  were  given  the  opportunity  to  review  the  Lessons 
Learned  Systems  write-ups  of  Chapter  IV  systems  and  concurrence  was  obtained  (Key 
Codes  I,  4,  5,  6,  8,  9,  II,  12,  13,  14)  in  all  cases  except  where  there  was  no  response 
(Key  Codes  2,  3,  7,  10  &  15).  No  attempt  was  made  to  obtain  concurrence  with  findings 
regarding  the  analysis. 


C.  RECOMMENDATIONS  FOR  FUTURE  RESEARCH 

There  are  a  few  avenues  for  future  resource. 

1.  The  number  of  Lessons  Learned  Systems  could  be  increased  to  either 
support  the  findings  of  the  analysis  (Influence  Diagrams)  and/or 
expand/revise  the  Handling  Variable  Set. 

2.  The  other  characteristics  such  as  Acquisition  could  be  transformed  into 
Variable  Set  such  as  an  Acquisition  Variable  Set  and  the  Influence 
Diagrams  could  be  expanded/revise  to  include  any  influences. 

3.  The  Handling  Variable  Set  and  other  Variable  Sets  and  environmental 
effects  could  act  as  input  into  an  equation  whose  computed  value 
corresponds  to  a  level  of  success  of  one  of  the  three  tasks  (receiving 
lessons,  quality  of  lessons  and  implementation  of  lessons).  The 
coefficients  of  the  equation  is  what  is  to  be  determined. 

4.  Key  Code  13,  the  International  Space  Station  Lessons  Learned  Database, 
probably  has  the  best  methods  of  Lessons  Learned  implementation  and 
would  be  a  good  case  study. 

5.  Although  there  are  other  good  Lessons  Learned  systems  in  Chapter  IV, 
Key  Code  8,  Department  of  Energy  Project  Hanford  Lessons  Learned 
System,  would  be  a  good  case  study  for  overall  operations. 
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D,  CONTRIBUTION  TO  THE  BODY  OF  KNOWLEDGE 

The  first  purpose  of  this  thesis  was  to  inerease  the  body  of  knowledge  that  exists 
for  Lessons  Learned  Systems.  This  thesis  has  eontributed  to  the  body  of  knowledge  that 
exists  for  Lessons  Learned  Systems  by  first  providing  information  regarding  the 
meehanics  and  experienees  of  fifteen  existing  Lessons  Learned  Systems.  This 
information  found  in  Chapter  IV  was  provided  by  Lessons  Learned  System  point  of 
eontaets  who  have  a  high  level  of  experience.  Although  specific  questions  were  asked 
and  answered  relating  to  the  analysis  section  of  this  thesis,  the  point  of  contacts  also 
provided  benefits  of  their  experience  on  the  subject  of  Lessons  Learned  System  design  or 
architecture,  particularly  Key  Codes  8  &  11.  This  collection  of  material  can  provide 
future  designers  of  Lessons  Learned  Systems  a  resource  to  review. 

The  second  contribution  of  this  thesis  is  a  further  development  of  Lessons 
Learned  System  decomposition.  It  has  expanded  the  Handling  characteristic  into  a 
Handling  Variable  Set.  The  Handling  Variable  Set  is  tool  that  can  compress  a  large 
quantity  of  qualitative  data  such  that  it  remains  encompassing  but  allows  multiple 
Lessons  Learned  Systems  to  be  compared  with  less  effort.  The  development  of  the 
Influence  Diagrams  has  also  advanced  the  decomposition  by  providing  a  cause  and  effect 
relationship  between  the  Handling  Variable  Set  and  the  three  major  tasks  of  a  Lessons 
Learned  System.  This  thesis  has  further  decomposed  one  characteristic  of  a  Lessons 
Learned  Systems  and  provided  a  methodology  to  decompose  the  other  characteristics. 

The  third  contribution  to  the  body  of  knowledge  is  that  it  has  provided  some 
evidence  that  supports  the  principles  expounded  in  Chapter  IT  One  principle  of 
knowledge  management  is  the  importance  of  management  support.  The  importance  of 
management  support  was  cited  a  few  times  in  the  success  of  a  Lessons  Learned  System. 

Knowledge  management  principles  also  cited  the  importance  of  being  pro  active 
in  the  collection  of  lessons  and  the  potential  difficulties  in  requiring  a  worker  to  perform 
a  task  and  record  lessons.  Existing  Lessons  Learned  System’s  experience  supported 
these.  Lessons  Learned  Systems  that  were  passive  had  concerns  about  receiving  lessons. 
Those  that  required  worker  development  of  input  as  part  of  the  task  cited  that  success  in 
receiving  lessons  was  very  dependent  on  management  enforcement  of  the  requirement. 
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Knowledge  management  prineiples  also  provided  numerous  suggestions  for 
transforming  an  experienee  into  a  quality  form.  This  has  been  reeognized  by  Lessons 
Learned  Systems  by  the  fact  that  practically  all  existing  Lessons  Learned  Systems 
incorporate  some  editorial  review  prior  to  dissemination.  The  fact  that  the  information 
provided  in  Chapter  IV  did  not  give  specific  details  of  the  review  could  be  interpreted  as 
the  need  for  the  editorial  review  to  be  tailored  to  the  specific  lesson  and  organization. 

The  principles  of  Organizational  Learning  were  also  empirically  supported  to 
some  degree.  Those  Lessons  Learned  Systems  that  were  strong  in  the  implementation 
task  used  methods  that  it  could  be  argued  promoted  tacit  learning  from  the  explicit.  The 
use  of  policy  implementers  to  write  lessons  prior  to  dissemination  required  tacit 
knowledge  to  be  understood  by  the  policy  implementers,  the  key  to  organizational 
learning.  Also  the  task  of  requiring  a  response  from  Lessons  Learned  targets  required  a 
tacit  understanding  to  begin. 

The  thesis  as  a  whole  contributes  to  the  body  of  knowledge  of  Lessons  Learned 
Systems  by  providing  examples  of  Lessons  Learned  Systems,  a  method  to  code  a 
characteristic,  by  the  Handling  Variable  Set  in  this  case,  and  some  work  in  the  cause  and 
effect  of  operational  choices  to  the  tasks  of  a  Lessons  Learned  System  through  the 
Influence  Diagrams.  As  the  importance  of  Lessons  Learned  Systems  become  realized, 
designers  can  refer  to  this  thesis  as  a  resource. 
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